On Metrics & Myths or Your Facts are From the Land of Make Believe

Its been a quiet week in Lake... Oh wait.  I'm not a famous radio personality with a show centered on a town that does not exist.  I'm a tester.

Sometimes though, I feel less like someone from Lake Wobegon, MN and closer to someone from Brigadoon.  Both are fictional, mythical if you will, and both have certain charms and appeal about them.  Except for one minor point.  Out of context, they make very little sense. 

So, the last several weeks I have been working away on studying metrics and concepts around them and things of that ilk.  The cause of that was the combination of "training" required by the day-job, and getting the new set of metrics for the "Scorecard" - yup - Metrics applied to the individual, team, group and department.  Oh my.

So, I went digging though my notes and found a variety of ideas, some good and some less than good, from a variety of sources, some reliable and some less than reliable.  Some of these we just plain contradictory.  Some had ideas that, in and of themselves seemed reasonable, until you considered the assumptions and presumptions that must be made and taken in for the numbers to actually make sense.

I found myself rereading articles by Cem Kaner, Doug Hoffman and others cautioning against misusing metrics.  I likewise found learned discussions around how metrics can be relied on if you take emotion out of the equation and look just at the hard, empirical data.

Then I saw a tweet from Michael Bolton, recommending the writings of  Laurent Bossavit as being worthy of  consideration.  So, I followed the link and began reading.  What I found was a fellow who had written an e-book that seems interesting.   Don't take my word for it.  His Twitter handle is @Morendil.  Search for him and begin reading.  Or, check out his e-book - Cool title - The Leprechauns of Software Engineering.  Find it here:  http://leanpub.com/leprechauns  You may not agree with everything, but much is worth your consideration.

Where was I?  Oh yeah.  Metrics.

Matt Heusser and I had an interesting chat last month while on a flight to New York.  He asked me my view on metrics.  I responded that my general view was that most people misuse the term and the concept. 

I believe that metrics should serve to address questions we are seeking enlightenment on (kind of like testing, no?)  A painfully large number of companies focus on stuff that is easy to count, without looking to see what that information might tell them - beyond the obvious.

I believe that most people trying to address questions with these "metrics" really don't have a good idea what the questions they want to ask are - and so they settle for what they can get easily.  Things like bug counts, test cases, test cases executed per day, failure rates and things of that ilk.  Instead of looking for things to help constructively help their staff, their people, do their work better, it is easier to look for control metrics.

They'll misquote Drucker or Lord Kelvin or - heck - maybe they've just heard so many truisms (that aren't really true) that are misquotes that they accept them at face value - an awful lot of us do.  They'll look to change behaviors by making a big deal about metrics and ... well, stuff.  What they get may not be what they intended to get.

Be careful in dealing with metrics - not all is what they appear.

Be careful when playing with dragons for you are crunchy and good with ketchup.

Where Has Pete Been, or Time Flies When You're... really busy

I can not believe so much time has passed since my last blog post. 

OVER A MONTH! 

I have several in the works and will be posting one shortly after this - I promise. 

Since the last update, I've been crazy busy at the day-job.  I spent several days in New York with Matt Heusser hanging with some really smart people and talking about testing.  I made some new friends and chatted with people I knew by reputation and cyberly - but met in person for the first time.

One night I participated in a panel discussion hosted by the Financial Services Special Interest Group of AST - the Association for Software Testing.  What an interesting experience - lots of different view points and ideas around some seemingly simple questions. 

We had a great time.

I've also had some interesting conversations when we got back - lots of interesting things happening in testing all around the world, it seems.

I had, at one point, signed up as a "contingency instructor" for the AST's BBST Foundations Course - translated, there were folks signed up, a head instructor and two assistants, and I agreed to help out if one of them could not do the class.  As it is, I got an email this last week asking if I was willing to help out.  So, I'll be doing some online instruction. 

There have been a stack of problems facing day job stuff - projects that simply were not working right - lots of work there.  Did I mention that before?

Oh yeah, I had "unexpected oral surgery" shortly after getting back from New York and I'm like WAY behind in my writing.  Sorry folks - I'll get there - I promise.

I also have been trying to catch up on the music writing I've been trying to get done.

And of course, being a husband, dad and - Oh yeah, my neice got married as well. 

So, sorry for the backlog - I'll get caught up soon. 

I promise.

CAST 2012 Tracks, Workshops, Emerging Topics, Coaching And Deadline #1

CAST 2012 is fast approaching! 

Well, some folks may think that JULY is not all that "fast approaching" in January - HOWEVER - there are some important dates coming up.

The Call For Participation is up (here). There are three basic types of presentations:

Interactive Workshops (140 minutes);
Regular Track Sessions (70 minutes - at least 25 minutes for discussion);
Emerging Topics (20 minutes - at least 5 minutes for discussion);

The deadline for Regular Tracks and Workshops is January 16.   As I write this, that is FOUR DAYS AWAY! 

The information you need to know about submitting proposals is on the website at the link above. If you are a Thinking Tester, I encourage you to consider attending CAST. If you are interested in telling people about your ideas, I encourage you to consider submitting a proposal.

Now, if you think you might like to present, and are unsure of actualling presenting - the Emerging Topics track may be a solution.  These are 20 minute snippets - enough for you to present the core of an idea and answer questions.  The deadling for THOSE submissions is June 18.  You can submit through the same process the Regular Track and the Workshop sessions are submitted. 
 
Here's where it gets interesting.  If you are a THINKING TESTER - and help people become better testers - the Test Coach Camp is the weekend before CAST!

Test Coach Camp will be held at the same hotel where CAST will be held.


Matt Heusser wrote about it here. The official AST release and Call for Participation can be found here.

These folks said it better than I can.

If you are interested in helping testers do their testing better, which is what Test Coaching is all about, and in Thinking Testers - Look into both of these - THEN ACT.

This is going to be a great week.

Janus Part 2: Looking Ahead 2012

Last year was really incredibly busy for me.  Work stuff happened that was crazy hectic, then the "speaker" thing moved from 1st to 2nd gear. Then the broader community expanded a bit more - or maybe my awareness of it did.  The local testing group moved from "once in a while" to "monthly meetings".

What am I looking forward to in 2012?  That is an interesting question.

Personal/Professional Development

This is something that is kind of included in the following areas.  There is much I want to learn and much I want to participate in - to help learn and see how things work in the world and not just in the theory/idea stage.  Broadly, I want to "engage closer" in the local test community, the broader community of thinking testers, a variety of projects and meetups and... life.

So, a little more detail...

Projects, Writing, Work.

The day-job is the day-job.  Yeah, there will be some interesting projects there, including a cool mobile device project.  The projects I mean here are some interesting side-projects - things with other people.  There are some cool initiatives with Matt Heusser that are interesting.  Matt is a bundle of energy with lots of ideas.  I'm flying as his wing-man on some items coming up (more on that as they get closer on the horizon.)

I'm looking into doing more writing.  Writing is cool.  It is work I can do anywhere and still have some semblance of a home life.  Its funny - the more you write on a topic, the more you find there is to learn about on the same topic.  So, if I can continue the process of writing/learning/sharing - so much the better.  There are some magazines I've decided I want to approach about article submission - and follow-up on some contacts from late in 2011.  Look for more on this in the near future.

This past August, I expressed an interest in getting more involved in the EdSIG - Education Special Interest Group of the Association for Software Testing.  Aside from a few emails, I've been really lax in that.  I want to change that this year.  I'd like to become more involved with, and as a result learn more about, training software testers - helping them learn and think and grow as craftsmen.

Two interesting factors here - First is the announced Test Coach Camp the weekend before CAST - the Conference for the Association of Software Testing.  This year's conference theme is The Thinking Tester - its going to be cool.  The Coach Camp though is a new addition and it looks fantastic - a peer conference to talk about improving how we coach testers.  I'm excited about that.

The second interesting factor is I volunteered to be a "contingency" instructor (I kind of missed the rush of people signing up to be "assistant" instructors because I let non-work email slip a bit while wrapping a project) for the BBST Foundations Course this March.  I don't know if it will balance out, but I'm looking forward to it. 

Community

Aside from the education and learning stuff, there are other aspects I'm looking forward to engaging (OK, note, don't try and write something after watching "Office Space" and trying to sound "professional").  The local tester group is up and running - next meeting is in two weeks and on a topic that should bring about good discussion - we're talking metrics.  Oh yeah, pass the popcorn. 

Toward the end of 2011, I began attempting to be more helpful in online forums - I'm afraid some folks consider my approach "annoying" - others seem to realize I'm asking questions to help me form an answer, and will "play along". 

There are, of course, a pile of conferences out there - the mentioned CAST conference in July.  STPCon Spring, I'm afraid I can't make work, but there are other possibilities out there.  It will be a busy, fantastic year that way, I think.

Cool People

There are many folks out there that I spend time with as much as I can.  There are others that, if I can, I want to spend more time with, learning from, learning with, sharing ideas and getting a clue.  Now, some are broadly spread - over North America, Europe, some in India... So, kind of a "I'd like to and I'm not sure I can make it work, but I want to" list. 

In generally alphabetical order, people I really want to find a way to hang with and learn from this year...

Perze Ababa   Ajay Balamurugadas   Bernie Berger   Paul Carvalho  
Salena Delesie   Markus Gaertner    Paul Holland     Phil Kirkham  
Micahel Larsen   Darren McMillan   Catherine Powell   Mark Tomlinson
And of course, the folks I enjoy learning from already, although meetings are sometimes few and far between, and often only by Skype - Fiona Charles, Michael Bolton, James & Jon Bach, Griffin Jones, Nancy Kelln, Lanette Creamer, Matt Heusser, Mel Bugai, Lynn McKee... and all the rest.

For all you have done to help me learn, thank you.  I look forward to learning with you all this year.

Janus Part 1: Looking Back 2011

One of the benefits of taking four years of Latin is that you pick up all kinds of interesting things that many other folks may miss.  Then again, an awful lot of people don't worry too much that "i.e." is an abbreviation of "id est" or, "that is".  Just like "etc." is an abbreviation of "et cetera"  - even though they may even SAY et cetera, I wonder how many know what it means?  I'm mean enough to not say here, and say "look it up" - unless you remember your Latin as well.

Janus, for whom the month of January is named, looked both forward and back.  That is a bit of what I want to do with this post and the next.  The post I wrote last night was a precursor to these couple of posts, partly because the things described yesterday laid the foundation for this past year and the year to come.

What I wrote January 1, 2011:

The Road Ahead...

The interesting thing is I've been thinking about the future. Well, not THE future, but what lay ahead for me professionally and how that may impact the family. It would seem there are several items that are possibilities for the coming year. One path would be to look for new work opportunities, either as a contract/consultant or as a full time, permanent employee. Yeah, as if "permanent" means much.

A bunch of folks commented privately, "Dude, pretty gutsy to say you'll be looking for work when you're still employed."  What I could not say then, was that in December, the entire staff of the company I worked for was told, in essence, that the company leadership was negotiating the sale of the company.  We did not know to which other company, nor what the terms would be.  Many of us speculated that the only reason we were told at that point, was because they needed us to sign releases of our stock options in case the sale closed before the end of the year. 

It was not a bold move to make such a prediction - I simply knew there was a likelihood that I'd be looking for work. When one company assimilates, well, acquires, another company, "long term employment" prospects for the staff of the acquired company are not terribly high.

As it was, I was not let go.  I was retained.  One colleague resigned after accepting a new position.  His last day, we had a farewell luncheon for him.  By the end of the next day, myself and another tester were all that remained of  our team.  One other person, a developer, had been transferred from the development staff to testing.  although others on the team

We are continuing, and moving forward.

Community

Another option is to become more involved in the testing community. Actually, I started working on that as well in 2010. What I mean is that reading blogs other folks write is a good way to learn what they're thinking is. Reading and participating in on-line forums is another way to both learn and become involved. Well, doing that as much as I can right now.

Of course, more actively engaging in both of these types of activities is on my list of things to do this coming year. Ya know, the funny thing is, the more I talk with folks about things I learn and have learned, the more I learn myself.

This continues.  I've been writing.  Alot.  STP Magazine and TechTarget's SearchSoftwareQuality both have run articles I've written.   my  more in my blog, and more engaged in forums than ever before. 

I expect this to continue and grow in the coming year.  That would be way cool.
 

Local Testing Groups

Another thing, the local testing group, GR Testers, has been going in fits and starts for a while. Meetings have been sparse of late. The most recent one, December, was kind of fun. There were a bunch of us sitting around a table, lots of wings, good beer and folks talking about testing. Good way to spend an evening. There's another meeting coming up Monday, 3 January. It makes it the first time in quite a while that there were back to back monthly meetings. Normally, they are officially held every other month. It seems that as more people are showing an interest, the meeting frequency will pick up.

I wonder how many other local testing groups are out there that have a meeting schedule based on "whenever" instead of "We meet at this time, and here are the next couple of topics we're focusing on at these meetings..." I believe that the more people know about local groups, the more they are invited to participate and the more information that is available about them, the more active and the stonger the community there is.

I think that pretty well sums up what I'm looking to do with the local group. I believe that getting more people involved and talking about testing is vital to improving not only our individual tradecraft, but the abilities of the local community. Sharing well reasoned ideas can do nothing but good, presuming all are allowed to learn and ask questions

The GR Testers, the local testing group, is up and running strong. The group has met monthly since that January post.  I've made it to most of the meetings. The ones I missed, I was out of town, usually at a conference. Cool.

Personal Development

Now, I realize that any of the above activities can lead to improving any individual participating. What I mean here is something a bit more. I had been signed up for the BBST Foundations course offered by the Association for Software Testing for a session in in the fall of 2010. Things happened and that session was cancelled. I could not take the session offered as an alternative.

The GOOD news, for me, is I am signed up to take the Foundations course this spring. YEAH! I am really looking forward to this. Everyone I know who took the course raves about it. Big-time excited.

I've continued reading blogs and articles and books and talking with people and... everything else. My goal is to continue learning and to continue to share what I learn.

For conferences, I'll be attending and presenting at STPCon in March in Nashville. I bought myself a birthday present and renewed my AST membership in October. If I can work it out, I'll be attending CAST in August in Seatle.

This happened beyond my wildest dreams.  I took and passed the BBST Foundations course.  Then, even though the schedule did not permit me to take the Bug Advocacy course - that is on the list for next year for me.  I also took the Instructor's Course from AST.  We'll see how the schedule works out this coming year.

Conferences.  I presented at STPCon (Spring) in Nashville.  I gave a joint presentation with my (then) boss, Kristin Dukic, as well as a presentation and lightning talk on my own.  I then was flattered, and honored, to attend and participate in CAST.   With Matt Heusser, I helped organize the Emerging Topics track, where a self-organized group selected topics submitted via a wiki - then ran for 20 minutes, every 25 minutes.  It was astounding.

After CAST, I had the opportunity to present at STPCon Fall in Dallas.  Matt Heusser and I did a day-long workshop (excerpts are on the Software Test Professionals site, under Podcasts) - then a joint track session on "Complete Testing".  THAT was a lot of fun. I also presented a track session on my own as well as a lightning talk.  Matt just gave a keynote.

Then since I was not busy enough, I presented at TesTrek in Toronto in November. 

Whew.

Other Stuff

Scads of people have encouraged me this year.  Among them, Matt Heusser, who put me in contact with the folks at TechTarget, and made the case that he could not do Emerging Topics at CAST on his own - which is how I got in.  Cool, heh?  THEN - Matt had so much fun with that, he asked if I'd be interested in doing a joint workshop in Dallas.  Oh yeah.  The interesting thing is that he's really a nice guy - as the folks who know him will attest. 

Also - Fiona Charles is supportive and encouraging.  She is really an amazing person who is willing to offer suggestions and ideas on how to improve articles, presentations, whatever.  She also is way cool.  She was one of the very first people that I consider a "Name" in testing, to ask for comments on a paper - the list me in the acknowledgements.  Humbling. 

Catherine Powell whom I met in person at STPCon in Nashville always has encouragement and good suggestions.  Michael Larson is a great guy.  He's got a great outlook on life and testing.  His blog is inspiring.  Doug Hoffman was the Head Instructor for the BBST Foundations course.  What a smart guy.  Nice as the day is long.  We had several very nice chats both at CAST or at STPCon Fall.  If you get a chance to see him present - DO.  Cem Kaner - yes DOCTOR Kaner - the drive behind the BBST Courses.  An ongoing inspiration.

There are more - Michael Bolton, Lynn McKee, Griffin Jones, Nancy Kelln, and many more.  These are the people I look to for inspiration and mental reinvigoration.

And of course, my lady-wife, Connie. 

I do not know what the future will bring.  I will discuss what I hope for the future in the next post. 

Rising From the Ashes or Finding Motivation in Disaster

This has been an interesting year.  There have been many fantastic things happen this year that at times it seemed like I was an observer, and not the one participating.  I've presented at more conferences this year than I attended any year before this.  People write emails asking questions, looking for insight or help with a sticky problem, as if I'm an expert. 

I've written before about not feeling like an expert.  This is not about that. 

While preparing for STPCon this past October I had an interesting in a couple of thoughts.  While working on the presentation, and a couple of papers, I mentioned one of these thoughts to a fellow member of the GR Testers group.  We chatted (cyberly) for a moment on how failure can be a great motivator.  We talked about people who had overcome problems and adversity to rise to great things.

Of course, there are also many examples of people who break under adversity.

I don't know what the differences are in those scenarios.  I don't know why some people crumble, others recover and come back to where they were and others rise to greater success than they have ever known.  The last group, to me, resembles space capsules, like the old Apollo capsules, that would whip around the moon to accelerate even faster than they were going.  Yeah, in Star Trek Kirk did the same thing with the Enterprise around the Sun.  Cool, no?

The second group, I kind of think of as being a bit like a rubber ball.  Not a fancy "Super Ball" that used to be sold with the assurance that it would bounce higher than where it was dropped from (and rarely did as far as I know) but a plain bouncing ball.  Comes back to where it was, but somehow not quite the same.

The first group, like I said.  I don't know why people fail to recover.  The just don't for a variety of reasons. 

Me.  Hah.  I was moving up.  I had left one company where I was simply unhappy, and joined another company as a Test Lead.  There were "issues" there.  I was hired to improve testing and change the way testing was being done.  Well, things were not working out.  I had a series of "those meetings" and the last one was handing me a package and me walking out the door.  (I'll be happy to give more details over adult beverages sometime, if you really want to know.)

So, I went home, popped in a video, cracked an adult beverage and said "What happens next?" 

Short term, I knew what had to happen - I needed to get ready to teach drum lessons that evening.  So, I had a single beer, watched a movie, fried some bacon and eggs and felt sorry for myself for 3 hours.  Then I made a strong pot of tea because I had work to do. 

I made a list of what I was good at and what I was not good at (no PC here, not right then.)  I went through the list of what I was good at that and highlighted those I liked to do and those I wanted to get better at doing. 

I then went through the list of what I was not good at. I split that list into "so what?", "consider improving" and "fix it".  I then considered a list of things I had read about and had done very little with or knew very little about.  I also made a list of things I knew nothing about, but I'd seen mentioned in articles and blog posts and said "this might be worth looking into." 

I then went on and read what I could, learned what I could and did some serious soul-searching on what I really wanted to do.  I then looked at how I would fix the stuff I really needed to fix.  This was hard - really, really hard.

This led me to the next step - Updating the resume, looking at what I wanted to do and where I wanted to do it.  I knew that (at the time) West Michigan was not a hot-bed for top-flite testing jobs, project management jobs and my development experience was not in technology that was in demand.  On top of that, the economy was beginning its downward slide.  So, I figured it would be a good likelihood that I would need to relocate. 

I looked and I looked... and I looked some more.  One month, I applied to 158 jobs. All over the US, Scotland, Ireland and Australia.

I learned a lot.  I've been applying those lessons ever since.

First - Be involved.  Online, locally, within the company, within the team.  Look for ways to learn and improve.  If someone looks for advice, guidance or a sympathetic ear - do what you can.  If something sounds familiar to a situation you were in, talk with them about your experience.

Second - Share.  Now, in some ways, this is similar to the first lesson.  Write.  Blogs, forum posts, responses to posts or online articles.

Third - Learn.  Keep learning, keep reading, keep thinking.

Four - Dare.

Five - Repeat.

Four years ago, the foundations for these really, really simple ideas were where I started.  I landed a job after a stack of interviews.  Some I knew would not be a good fit.  Others, well, they decided it would not fit.  I was ok with that.  When I landed the gig I landed, I talked with people. I learned.  I learned their applications, their methods and their personalities.  I learned how they worked and did things. 

I shared ideas and experiences. I contributed when I could and asked questions when I did not understand. 

Then people began asking me questions - How can we learn more about... Have you ever run into...

As a result of one series of these conversations, I landed at TesTrek in Toronto, where I met Fiona Charles and Michael Bolton in person, for the first time.  I also met a whole slew of people I had never met before, Nancy Kelln, Lynn McKee and slew of other bright folks. 

That week in Toronto resulted in me getting more involved, helping revitalize/reinvigorate the GR Testers, then scrap my drumming blog and move to writing on testing.  That helped with presenting at conferences... and that led to, well, this most astounding year.

Where did this come from?  Getting fired.

You don't need to get fired/sacked/down-sized/happy-sized/whatever to do the same.  If you want to grow, then do it.  If you want to get involved, do it.

The fact is, doing these things may not make you a leader or a superstar or being called an expert.  But, if the world comes tumbling down around you, if you have been doing these things, others can step up and help.  If you have established connections and a reliable cadre of people, they can help just as you can help them.

On Patterns and Blinking and Puzzles and Expectation

Our family has a lot of traditions around the winter holidays, Christmas and New Year.  One tradition is working on a massive 1,000+ piece jigsaw puzzle.  We see it as beneficial in many ways.  When the kids or grandkids are around, for them to participate ("become engaged") this thing that we are doing, they need to slow down.  I've yet to take any pleasure from assembling a puzzle that can be whipped through in an hour or two.

Our puzzles tend to take a week or more to be completed.  We'll start them one evening, then each day tinker a bit as each person has a few moments.  In the evening, we try and set aside 30 or 40 minutes to work on the puzzle together.  We've found it a great extension of "dinner table conversation" where we get caught up with each others' day. 

Oh, we both like doing puzzles too, which is perhaps the biggest reason why we do them.

So, this year's puzzle was a photograph of a Scottish castle, no I don't know which one, with hills and mountains and things in the background, a bit of water near the castle (hard to tell if it is a river or a loch or merely a fair sized pond.)  Like a lot of the better, or harder, puzzles, there were many bits that, well, looked a lot like other bits. 

In sorting out which bits are which, you need to look for subtle differences - small changes or variances in the overall image.  So, this last week, I had a portion that I was sure were part of the castle's battlements - the tops of the walls or towers.  Then I noticed another piece - JUST like the one I had in my hand. but a little different.  There was a small line in the piece I had that was not in this new piece.

I blinked.  Literally.

The portion I was working on was indeed part of the battlements - but the reflection of the battlements in the water - not the actual "top of the wall" stuff.  I was reminded of a defect I had spent time trying to track down on a recent project. 

I had a set of expected results and behavior, my "oracles" - and the results - what I was actually seeing, were really really similar, but not quite what I was expecting.  It looked right, but something did not feel right.  What I was expecting, based on the described behaviors and expected results, was generally what I was seeing. But something did not feel right.

It was kind of like the puzzle pieces.  One looked like what I expected it to look like.  The other was, well, different. 

That got me thinking about other things. 

How many times are we certain that what we expect is really what we should expect?  Is it not possible that the expectations are the "bugs"?  What is it that makes the "expected results" "right"?  Even when you are the one who created the "expected" results, how well do you really understand the software?  Do you have a certain understanding as to what the changes will result in? 

In my case, my "expected results" were what was at fault - both in the puzzle and the testing. 

Once I realized my mistake in the testing, it became much easier to move forward.  I will never know about the puzzle, I'm afraid.  The orange tomcat who lives in the house with us decided that he had enough of us assembling the puzzle.

I believe, but am not certain, that we found all the pieces after he scattered them from the table.

Should we try and put that puzzle together in the future, I expect we'll find out about any missing pieces.