On Metrics & Myths or Your Facts are From the Land of Make Believe

Its been a quiet week in Lake... Oh wait.  I'm not a famous radio personality with a show centered on a town that does not exist.  I'm a tester.

Sometimes though, I feel less like someone from Lake Wobegon, MN and closer to someone from Brigadoon.  Both are fictional, mythical if you will, and both have certain charms and appeal about them.  Except for one minor point.  Out of context, they make very little sense. 

So, the last several weeks I have been working away on studying metrics and concepts around them and things of that ilk.  The cause of that was the combination of "training" required by the day-job, and getting the new set of metrics for the "Scorecard" - yup - Metrics applied to the individual, team, group and department.  Oh my.

So, I went digging though my notes and found a variety of ideas, some good and some less than good, from a variety of sources, some reliable and some less than reliable.  Some of these we just plain contradictory.  Some had ideas that, in and of themselves seemed reasonable, until you considered the assumptions and presumptions that must be made and taken in for the numbers to actually make sense.

I found myself rereading articles by Cem Kaner, Doug Hoffman and others cautioning against misusing metrics.  I likewise found learned discussions around how metrics can be relied on if you take emotion out of the equation and look just at the hard, empirical data.

Then I saw a tweet from Michael Bolton, recommending the writings of  Laurent Bossavit as being worthy of  consideration.  So, I followed the link and began reading.  What I found was a fellow who had written an e-book that seems interesting.   Don't take my word for it.  His Twitter handle is @Morendil.  Search for him and begin reading.  Or, check out his e-book - Cool title - The Leprechauns of Software Engineering.  Find it here:  http://leanpub.com/leprechauns  You may not agree with everything, but much is worth your consideration.

Where was I?  Oh yeah.  Metrics.

Matt Heusser and I had an interesting chat last month while on a flight to New York.  He asked me my view on metrics.  I responded that my general view was that most people misuse the term and the concept. 

I believe that metrics should serve to address questions we are seeking enlightenment on (kind of like testing, no?)  A painfully large number of companies focus on stuff that is easy to count, without looking to see what that information might tell them - beyond the obvious.

I believe that most people trying to address questions with these "metrics" really don't have a good idea what the questions they want to ask are - and so they settle for what they can get easily.  Things like bug counts, test cases, test cases executed per day, failure rates and things of that ilk.  Instead of looking for things to help constructively help their staff, their people, do their work better, it is easier to look for control metrics.

They'll misquote Drucker or Lord Kelvin or - heck - maybe they've just heard so many truisms (that aren't really true) that are misquotes that they accept them at face value - an awful lot of us do.  They'll look to change behaviors by making a big deal about metrics and ... well, stuff.  What they get may not be what they intended to get.

Be careful in dealing with metrics - not all is what they appear.

Be careful when playing with dragons for you are crunchy and good with ketchup.

Measurements and Metrics, Or How One Thing Led to Another

So, once upon a time, my dear daughter and her beau gave me a combination "Christmas and New Job" present.  Yeah, I was changing jobs in late December... What was I thinking?  Not sure now, but it seemed like a good idea at the time. 

Anyway, this gift was an M&M dispenser.  Yeah. Pretty cool, eh?  Turn the little thingie on the top and a handfull of M&Ms would fall through a little chute and come out the bottom.  Not too shabby!

So, move along to the summer of 2008.  The company I was working for had a huge, big, ugly release coming out.  It was the first time with a new release process and schedule and nerves were pretty thin all the way around, developers, testers, support folks, bosses, everyone.  Well, being the observant fellow, I realized that we were consuming a LOT of M&Ms - Of course, it helped that the dispenser was at my desk, in my humble cube/work-area.

So, I started keeping track of how much candy we went through.  The only folks who partook of these multi-coloured delicacies were the QA/Tester group and a couple of brave developers who realized that we were not contagious and they could not catch anything from us.  (They also learned that they might learn something from us and we testers might learn something from them.)

What I discovered was kind of interesting.  As the stress-level went up, so did the consumption of M&M's.  As things were going better and things were looking good, then consumption went down. 

Using a simple Excel spreadsheet, I added up the number of bags eaten (it helps that they have the weight on them) as well as the partial bags each week.  Then using the cool graphing tool in Excel, I could visually represent how much we went through.  By correlation, the level of stress the team was under.

After about a month, I "published" the results to the team.  SHOCK!  GASP!  We went through HOW MUCH??????

Then the boss sat down with me and looked at the wee little chart.  "What was going on during this week?"  Ah-HA!  The first obvious attempt to match what the graph was showing.  I tracked usage for the rest of the year.  The amount the team consumed over the six months or so that I tracked, lined up remarkably with due dates and, interestingly, defects reported in testing. 

One thing led to another, and the dispenser was put away for a time.  In mid-2009, for reasons which now I don't recall, the M&Ms came back out.  As the crew realized this, consumption went up.  And up.  And up.  Eventually, I noticed that the same pattern demonstrated before was coming back. 

I learned two things doing this exercise (which I continue to do.) 

One, is that it is possible to measure one thing and be informed on another.  Now, I am well aware of the First and Second Order (and other) Measurements described by some of the great ones in our craft.  This exercise brought it home to me in ways that the theoretical discussions did not. 

The other thing, sitting at a desk and making a meal of M&M's is a really, really bad idea.

Music, or Testerman's Ramble

At one point in my life I played in a band that performed a variety of Irish traditional and folk music.  We also played a fair amount of Scottish traditional and folk as well, however, it seems if you play or sing a single Irish song, you are labelled "Irish" and you'll be crazy-busy in March, and pretty slow the rest of the year.  Unless you work really hard and play reasonably well.

So a side-effect of playing in a band that performs this stuff  is, when you get good enough for people to pay you money to go to their towns, cities, festivals, whatever, you will run into other folks who play the same type of music.  When schedules permit, this often devolves into a session / sessiun / wild-music-playing party.  There are certain protocols that most folks follow in these events - and the fantastic thing is that usually the level of play is quite good.  Tempos are snappy so reels drive forward and hornpipes can be lilty (and tend to run around Warp 9) and jigs are of a nature where feet rarely touch the ground. 

Now, these uber-sessions are not so different than more traditional ones held in houses or coffee-shops or bars or clubs.  The big difference is the recognition that there are no light-weight players and everyone has mastered their craft.  This is not always the case at other sessions. 

I have been out of the performing trad/folk music for several years now, and in the last year began attending some of the local sessions, just to get my feet wet.  I was a bit rusty on bodhran, the Irish hand frame drum, which I had played for 20 years on stage and in sessions.  My limited ability on penny whistle was nigh-on vanished - I remembered tunes and could call phrases from my memory to my finger tips, but I'm effectively starting over.  With crazy work and home schedule it has been hard to find time to practice , let alone become "street legal" on whistle.

So, I show up at the Sunday night sessions and play a couple of tunes on whistle when they come up.  I will also play the bodhran a bit, depending on the number of people there (it does not take many drums to become "too many" for the melody instruments - whistles, mandolins, fiddles, flutes, dulcimers and the like.) 

This last Sunday there were a fair number of players.  There were 8 or 9 "melody" players, a couple of guitars, a tenor-banjo, who played melody when he knew the tune and vamped when he did not - and me on drum (with the occaisional contribution of bones.)  Some of the players are quite experienced and I have seen around for many years.  Some are between beginner and novice.  Some are "in between" levels of experience. 

One tune in particular would have made me stop the band, if it was a "band" that was playing and have them start again.  That typically isn't done in sessions - so I did the "drummer equivalent" and simply stopped playing.  One of the mandolin players, who knew me and has also been around the block gave a little smile and he stopped as well.  We were treated to a rare sight of 6 people who were absolutely certain what the "correct" tempo was for the tune that was being played - and none of them would give an inch - or a click on the metronome.  The guitar players seemed to play along with which ever melody instrument was close to them and generally the best description was "trainwreck."

That reminded me of a projet I had worked on some time ago.  I was not on the project originally, but was brought in as part of a desperation move to fix it.  Like in the tune on Sunday, each of the participants knew what the right thing to do was.  The problem was none of them agreed on what that thing was.  "Blood on the Green" was an apt summation of that effort.  The programmers were berated for not following instructions - but how do you follow instructions when there are multiple, conflicting sets of instructions? 

Because of the "political nature" of the project, no managers or directors were willing to step up and take on some form of leadership role for fear that there would be repercussions for doing so.  The PM, BA and Dev Lead floundered without some form of direction from their respective management teams.  Information was contradictory at best. 

In the end, a Director put his foot down, asserted control and forced the issue.  Me being added to the project was part of forcing the issue.  Until that point, the uncertainty of the leadership was sapping the ability of the project group to operate as an effective team.  Like the music session last week, no one had a clear picture as to what was "right" and where the center of gravity was. 

People can urge "Best Practices," "Standards," "Process" and "Metrics" all they want.  In some contexts, that may be the right thing.  However, wiothout a clear understanding of the intent of the effort, nothing will save the project.  Ulysses S. Grant, that prescient Software Oracle (well, American General turned President) warned that indecision was worse than a wrong decision.  Wrong decisions could be countered by "right" decisions, but no decision, from leadership, leaves your group floundering looking for a center.