On Metrics & Myths or Your Facts are From the Land of Make Believe

Its been a quiet week in Lake... Oh wait.  I'm not a famous radio personality with a show centered on a town that does not exist.  I'm a tester.

Sometimes though, I feel less like someone from Lake Wobegon, MN and closer to someone from Brigadoon.  Both are fictional, mythical if you will, and both have certain charms and appeal about them.  Except for one minor point.  Out of context, they make very little sense. 

So, the last several weeks I have been working away on studying metrics and concepts around them and things of that ilk.  The cause of that was the combination of "training" required by the day-job, and getting the new set of metrics for the "Scorecard" - yup - Metrics applied to the individual, team, group and department.  Oh my.

So, I went digging though my notes and found a variety of ideas, some good and some less than good, from a variety of sources, some reliable and some less than reliable.  Some of these we just plain contradictory.  Some had ideas that, in and of themselves seemed reasonable, until you considered the assumptions and presumptions that must be made and taken in for the numbers to actually make sense.

I found myself rereading articles by Cem Kaner, Doug Hoffman and others cautioning against misusing metrics.  I likewise found learned discussions around how metrics can be relied on if you take emotion out of the equation and look just at the hard, empirical data.

Then I saw a tweet from Michael Bolton, recommending the writings of  Laurent Bossavit as being worthy of  consideration.  So, I followed the link and began reading.  What I found was a fellow who had written an e-book that seems interesting.   Don't take my word for it.  His Twitter handle is @Morendil.  Search for him and begin reading.  Or, check out his e-book - Cool title - The Leprechauns of Software Engineering.  Find it here:  http://leanpub.com/leprechauns  You may not agree with everything, but much is worth your consideration.

Where was I?  Oh yeah.  Metrics.

Matt Heusser and I had an interesting chat last month while on a flight to New York.  He asked me my view on metrics.  I responded that my general view was that most people misuse the term and the concept. 

I believe that metrics should serve to address questions we are seeking enlightenment on (kind of like testing, no?)  A painfully large number of companies focus on stuff that is easy to count, without looking to see what that information might tell them - beyond the obvious.

I believe that most people trying to address questions with these "metrics" really don't have a good idea what the questions they want to ask are - and so they settle for what they can get easily.  Things like bug counts, test cases, test cases executed per day, failure rates and things of that ilk.  Instead of looking for things to help constructively help their staff, their people, do their work better, it is easier to look for control metrics.

They'll misquote Drucker or Lord Kelvin or - heck - maybe they've just heard so many truisms (that aren't really true) that are misquotes that they accept them at face value - an awful lot of us do.  They'll look to change behaviors by making a big deal about metrics and ... well, stuff.  What they get may not be what they intended to get.

Be careful in dealing with metrics - not all is what they appear.

Be careful when playing with dragons for you are crunchy and good with ketchup.