Improving Processes, Part II

O would some power the giftie gie us
to see ourselves as others see us.

 - Robert Burns, To a Louse

Right. So if you need that translated, Rabbie was saying this, in English: 

O would some power the gift to give us
to see ourselves as others see us.

There are a couple of things that are just hard to do for most people.  One, is to change our perspective and see what others see in us.  The really hard part is similar to that: to look at ourselves honestly, and see what we really are like. 

Any time that self examination comes into play, most folks will avoid it as much as possible.  We can tell wee little fibs and stories and justify actions by many interesting machinations.  Yet when we strip those aside, what are we left with?

That is the really hard part in any form of process improvement.  When we look at ourselves, as individuals or as a group, the challenge is to set aside what we wish we were doing or like to think we do, and focus on our true actions.

If our real processes and the official process don't match, we need to ask ourselves, "Why not?" 

Indeed, to see ourselves as we really are, or as others see us, can be an event that we don't want to face.  Without doing that, we can never improve.

Improving Processes and Other Stuff, Part 1

I've been teaching a lot of pipe band drumming workshops lately.  Well, "a lot" compared to the last two years anyway.  They can be hard, but generally are great fun.  By the time I realize how tired I am, I'm half way home - close enough where a cup of Tim Horton's coffee will get me home (yes, there are some in Michigan.)

So, this last session was a mix of absolute beginners and those that were a step or two beyond that.  They all play in the same band, or aspire to anyway, and so have a common bond between them.  Part of the intention of the workshop organizers is to not only teach the beginners, but teach the more advanced players how to teach. 

That actually is easier than it sounds, at least with drumming.  I get to present an idea, work on some exercises, see who is getting it and who isn't.  If it is a physical thing, there are other exercises to try.  If it is a mental thing or thought process thing, then I present the same basic idea another way - changing the context sometimes makes it easier. 

This last session we were working on triplets.  Cool things, those triplets.  They are also bread-and-butter stuff for pipe band drumming. The kind of thing where if you don't get them, your future as a pipe band drummer is quite limited.  One guy was having a bit of a hard time with them than the other students.  Mind you, these students ranged in age from 8 years to the mid-30's or so.  This particular fellow was doing alright, but was having an issue with getting his head around the idea of "three notes where normally there are two."

I handed out a bunch of candy/sweets to the other participants and asked this fellow to play the bit he was having a problem with.  Perfect.  So I asked him to do it again.  Perfect.  Third time was still perfect.  Hmmm... it does not look like its the actual "hard bit" thats the issue.  So I had him play the exercise from the beginning.  Trainwreck!  Had him slow things down and do it again - same thing.  The second time I noticed something in what he was doing.  As he got closer to the "hard part" his grip tensed up (he gripped his sticks harder) his muscles in his forearms tensed visibly - both bad things for drummers.  Try as he might, the sticks simply were not going to do what he wanted them to do.  When he jumped right into it, things worked fine.

If the first step to solving a problem is recognizing you have a problem, how do you know what the problem is?  In this poor fellow's case, he knew he had a problem and simply could not see what it was.  It wasn't the problem he thought he was having - it some something else entirely.  When he stayed relaxed throughout the line of the exercise, he played it flawlessly.  Problem solved.  But what was the problem? 

When it comes to process improvement for nearly anything, I try and apply the same approach:  There may be a problem, lets see what the symptoms are and see if we can isolate the problem - instead of whacking the symptoms or results of the problem. 

When looking at Test Process Improvement in particular, the problem that gets described is usually a symptom or list of symptoms - not the actual problem.  We can stomp on symptoms one at a time, without really addressing the crux if the problem.  That will continue to churn and bubble and fester until something else breaks. 

The "problems" presented usually are presented in a handfull of ways:

• Testing is taking too long;
• Testing costs too much;
• Too many defects are being found by customers.

Of course, there are other variations, but what I have seen have usually falls into one (or more) of those complaints.

When I was younger and not as diplomatic as I am today, my reaction to each of those points generally ran something like "Compared to what?"  Now, in my more mellow state and place of being, I only think that.  Sometimes loudly, but what comes out of the mouth runs something like, "Can you explain what you mean by 'too long' so I can understand better what you expect?  An awful lot of the time our original estimates on test effort or duration are based on the understanding of what the project will entail at the time the estimates are made.  As the project develops, we learn more and that will impact both the revised effort estimates and the actual duration.  What we are doing is based on the instructions given to us and the mandates for what critical processes must be validated.  Perhaps this needs to be reconsidered."

So, I guess that's a long-winded version of "Compared to what?" but it tends to go over a bit better. 

What I do wonder, however, when a boss-type says something like those statements, is "Are those symptoms or problems?"  Are we running over timelines and cost estimates because of other things than lousy estimates?  Are "due dates" being missed because of testing?  Are there a flurry of defects keeping customers from using the new release? 

Is there something else going on and these are perceptions of problems, rarther than symptoms of problems?

Requirements, Traceability and El Dorado

The day-job has been crazy busy the last several weeks.  I have several half-written entries that I want to finish and post and with the project hours, and stuff needed to be done at home, there simply has not been much time.  However, I've been lurking in a couple of places, reading posts and email conversations, getting my fix of "smart people's thoughts" that way. 

The interesting thing is that a couple of themes have crept back up and I finally have the chance to take a look at the topic(s) myself and consider some aspects I may not have considered. 

The initial question revolved around defining requirements and establishing traceability of test plans, test cases and the like back to requirements.  By extension, when executing the test cases defects found should likewise be able to be traced back to said requirements.

Now, folks who have read my blog in the past will realize that I've been writing about requirements off an and for some time.  Well, actually, its more "on" than "off."  I write about requirements and testing a lot.  Possibly this is because the struggles of the company I work with on defining requirements and the subsequent struggles to adequately test the software products created from those requirements.  Now, to be clear, it is not simply this company I am with that has an issue.  Most places I've worked have been seriously "requirements challenged." 

One thing that sends up every warning flag the back of my neck has is the idea that we can fully define the requirements before doing anything else.  I know, Robin Goldsmith has an interesting book on defining "REAL requirements" and he has some good ideas.  In light of the shops where I have worked over the last, oh, 25 and more years, some of these ideas simply don't apply.  They are not bad ideas, in fact, I think testers should read the book and get a better understanding of it.  (Look here to find it, yeah, I know its pretty pricey - expense it.)

Having said that, how many times have we heard people (developers, testers, analysts of some flavor, project managers, et al.) complain that the "customers" either "don't know what they want" or "changed their requirements."  I've written before about the understanding of requirements changing, and how considering one aspect of a project may inform understanding on another.  When this happens in design, development, or worse testing, the automatic chorus is that the "users" don't know what they want and work will need to be changed.   All of us have encountered this right?  This is nothing new, presumably.

My point with this revisit is that if you are looking to find the cause of this recurring phenomenon, look in a mirror.  All of us have our own biases that effect everything we do - whether we intend to or not. 

So, if your shop is like some I've worked in, you get a really nice Requirements Document that formally spells out the requirements for the new system or enhancement to the existing system.  The "designers" take this and work on their design.  Test planners start working on planning how they will test the software and (maybe) what things they will look for when reviewing the design.

Someone, maybe a tester, maybe a developer, will notice something; maybe an inconsistency, maybe they'll just have a hunch that the pieces don't quite go together as neatly as they should.  So a question will be asked.  Several things may happen.  In some cases, the developer will be given a vague instruction to "handle it."  In some cases, there will be much back and forth over what the system "should" do, then the developer will be told to "handle it."

At one shop I worked at, the normal result was a boss type demanding why QA (me) had not found the problem earlier.

My point is, defining requirements itself is an ongoing process around which all the other functions in software development operate. 

Michael Bolton recently blogged on test framing.  It is an interesting read.   It also falls nicely into a question raised by Rebecca Staton-Reinstein's book Conventional Wisdom around how frames and perspectives can be both limiting and liberating. 

This brings me back to my unanswered question on Requirements:  How do you show traceability and coverage in advance when it is 99.99% certain that you do not know all the requirements?  Can it really be done or is it a fabled goal that can't be reached - like the city of gold? 

Wiser people than me may know the answer.

Learning and Teaching and Leading

One thing I learned early on when teaching drumming students, particularly beginners, is that the person who learns the most is often the teacher.

It never seems to matter whether the lesson is an individual or group lesson, focused on one style or general drumming - the process of teaching beginners forces the instructor to reconsider things that the instructor simply does.  This forces the teacher to reconsider all that he does, find interesting foibles or potential weaknesses, then correct or change them as needed for working with the student. 

The interesting thing is that this reflection sometimes leads to profound understanding of what the student is learning and what the instructor is conveying.  When preparing for the odd lunch-and-learn or training session at the office I never really had that kind of an experience - or when presenting such sessions. 

On Improvement...

This last couple of weeks something interesting happened.  I've been preparing a presentation on Test Process Improvement for TesTrek in October.  I wasn't scheduled to present, or lead a workshop, but as a couple of presenters had to cancel, Viola!  I'm on the presenters list.  Then, a couple of other things came into my observation. 

There have been several conversations on email lists I'm a participant in, as well as forums, on the dreaded M word.  Yes - Metrics.

On top of this, I had a remarkably revealing email conversation with Markus Gartner - amazingly bright guy.  This came about because the questions I submitted for the "Ask the Tester" were submitted after the magic number of 10 had been reached.  However, they were forwarded to Markus and that presented me the opportunity to learn and be rinded of things I once knew and had forgotten (or channelled off into a safe place in my memory.)

My question to Markus was centered on his take of "Test Process Improvement" in an Agile environment.  The bulk of his response was reasonably close to what I expected - in fact, reassuringly close to what I had prepared for the presentation so my confidence level increased dramatically in what I was saying.  (Yes, a little reassurance is sometimes a good thing, particularly when one is a very little fish hanging out with very big fish.) 

He had one idea that I did not have.  And it left me gob-smacked.  Tacked onto an already interesting sentence about the organization's management, Markus said "... or they don't trust testing anymore."

On Trust...

I was immediately thrown back many years to when Developers were called Programmers and when I was working as a COBOL Programmer on a large IBM mainframe.  I had a Manager who did not trust his staff.  Not because they were inexperienced, but because he simply did not trust them.  To this day, I do not know why that was the case.  I can surmise why, but it has little to do with the point.  Suffice to say, it was an un-happy work environment. 

Markus made an interesting observation.  His point was that in Agile, the very purpose is to engender trust amongst all participants. Additionally, when management is invited to observe the meetings, they can gain an understanding of what is being done by their staff and as their understanding increases, so to should their level of trust. 

When a group or a team has lost the trust of its management, the task of regaining that trust is nigh-on insurmountable.  Likewise, if a manager or lead has lost the trust of the group they are to lead or manage, the results will almost certainly be dire.

On Process...

Thus, when the call comes down for "better metrics" or "process improvement" or any other number of topics.  What is the underlying message?  What is it that someone is hoping to gain?  Do they know?  CAN they know?  Are they guessing? 

Much is debated around QUANTifiable and QUALifiable considerations, measurement and understanding.  I am not nearly bright enough to join into that fray fully-fledged. 

What I have seen, however, is when Managers, Directors, VPs, EVPs, and big-bosses of all varieties are looking for something - nearly anything will suffice.  A depressing number of times, I have seen management groups flail around what is wanted - then issue and edict announcing the new policy or practice or whatever it is.  These tend to roll-out like clockwork, every three to six months. 

Each company where I have worked that followed that practice engendered a huge amount of cynicism, resentment and distrust.  The sad thing is that these rather stodgy companies - including some that were quite small and prided themselves on having no Dilbert-esque Pointy-Haired-Boss behaviors - were wasting an amazing opportunity.

The first step to fixing a "problem" is figuring out what the problem is.  If there is no understanding over why policies or procedures are changing and no feed-back loop on the purposes behind the changed, will the average rank-and-file worker stand up and say "What do you hope to change/improve/learn from this?"  At some companies - maybe.  But I have seen relatively few times where the combination of policy-dujour and staff willing to stick their necks out and ask questions both exist in the same organization. 

On Leadership...

What I have learned, instead, is to look at all sources of information.  Explain what the problem or perceived problem is.  Ask for input - then consider it fairly.  To do so is not a sign of weakness - it is a sign of strength.  That the leadership of the organization have enough trust in their workers to approach them with a problem and work together toward a solution.

This, in my mind, is the essence of building a team. 

If you throw a bunch of people together without a unifying factor and expect great things it is silly in the extreme.  In the military, "Basic Training" serves this purpose - laying the groundwork to trust your comrades and follow the direction of officers and non-commissioned officers.  In the end though, the object is teamwork:  learning to work together using each persons strengths to off-set others weaknesses. 

Why is it that so many managers miss this rather elementary point?  For a team to "work" they must learn to work together.  If the Lead or Manager has not built the group into one capable of working together, like a team, what, other than professional pride, will get any results at all? 

Although I can not prove this, in a scientific method as it were, I suspect that it is the essence of the problem mentioned above.  The question I do not know the answer to, although suspect it, is the question of leadership in this instance. 

Is it that they, the leaders, have no idea how to build a team?  Is it possible that the step of instructing the fledgling team and shaping it into the needed form was too challenging?  Could it be that in the process of doing so, their own closely held beliefs, habits and foibles were more dear than the building of a successful team?

If this basic lack is present, does it contribute to the selection of what is easy over what is right? 

These are the ideas that have been floating through my mind while preparing the presentation and workshop lessons for the session at TesTrek.  If the master knows that he is but a beginner in the craft, what of those who consider themselves experts in all aspects of our trade.

Can this be at the root of the behaviours I've seen first hand and read about?  Are they feeling so insecure in their own abilities that they mistrust their own staff, the team they are charged with leading?  Is it to make up for this lack, they flounder and grasp for tips or magic revelations that will show them the "path?"  Is that why there is a continuing and perpetual drive for Metrics and Process Improvement?

Music, or Testerman's Ramble

At one point in my life I played in a band that performed a variety of Irish traditional and folk music.  We also played a fair amount of Scottish traditional and folk as well, however, it seems if you play or sing a single Irish song, you are labelled "Irish" and you'll be crazy-busy in March, and pretty slow the rest of the year.  Unless you work really hard and play reasonably well.

So a side-effect of playing in a band that performs this stuff  is, when you get good enough for people to pay you money to go to their towns, cities, festivals, whatever, you will run into other folks who play the same type of music.  When schedules permit, this often devolves into a session / sessiun / wild-music-playing party.  There are certain protocols that most folks follow in these events - and the fantastic thing is that usually the level of play is quite good.  Tempos are snappy so reels drive forward and hornpipes can be lilty (and tend to run around Warp 9) and jigs are of a nature where feet rarely touch the ground. 

Now, these uber-sessions are not so different than more traditional ones held in houses or coffee-shops or bars or clubs.  The big difference is the recognition that there are no light-weight players and everyone has mastered their craft.  This is not always the case at other sessions. 

I have been out of the performing trad/folk music for several years now, and in the last year began attending some of the local sessions, just to get my feet wet.  I was a bit rusty on bodhran, the Irish hand frame drum, which I had played for 20 years on stage and in sessions.  My limited ability on penny whistle was nigh-on vanished - I remembered tunes and could call phrases from my memory to my finger tips, but I'm effectively starting over.  With crazy work and home schedule it has been hard to find time to practice , let alone become "street legal" on whistle.

So, I show up at the Sunday night sessions and play a couple of tunes on whistle when they come up.  I will also play the bodhran a bit, depending on the number of people there (it does not take many drums to become "too many" for the melody instruments - whistles, mandolins, fiddles, flutes, dulcimers and the like.) 

This last Sunday there were a fair number of players.  There were 8 or 9 "melody" players, a couple of guitars, a tenor-banjo, who played melody when he knew the tune and vamped when he did not - and me on drum (with the occaisional contribution of bones.)  Some of the players are quite experienced and I have seen around for many years.  Some are between beginner and novice.  Some are "in between" levels of experience. 

One tune in particular would have made me stop the band, if it was a "band" that was playing and have them start again.  That typically isn't done in sessions - so I did the "drummer equivalent" and simply stopped playing.  One of the mandolin players, who knew me and has also been around the block gave a little smile and he stopped as well.  We were treated to a rare sight of 6 people who were absolutely certain what the "correct" tempo was for the tune that was being played - and none of them would give an inch - or a click on the metronome.  The guitar players seemed to play along with which ever melody instrument was close to them and generally the best description was "trainwreck."

That reminded me of a projet I had worked on some time ago.  I was not on the project originally, but was brought in as part of a desperation move to fix it.  Like in the tune on Sunday, each of the participants knew what the right thing to do was.  The problem was none of them agreed on what that thing was.  "Blood on the Green" was an apt summation of that effort.  The programmers were berated for not following instructions - but how do you follow instructions when there are multiple, conflicting sets of instructions? 

Because of the "political nature" of the project, no managers or directors were willing to step up and take on some form of leadership role for fear that there would be repercussions for doing so.  The PM, BA and Dev Lead floundered without some form of direction from their respective management teams.  Information was contradictory at best. 

In the end, a Director put his foot down, asserted control and forced the issue.  Me being added to the project was part of forcing the issue.  Until that point, the uncertainty of the leadership was sapping the ability of the project group to operate as an effective team.  Like the music session last week, no one had a clear picture as to what was "right" and where the center of gravity was. 

People can urge "Best Practices," "Standards," "Process" and "Metrics" all they want.  In some contexts, that may be the right thing.  However, wiothout a clear understanding of the intent of the effort, nothing will save the project.  Ulysses S. Grant, that prescient Software Oracle (well, American General turned President) warned that indecision was worse than a wrong decision.  Wrong decisions could be countered by "right" decisions, but no decision, from leadership, leaves your group floundering looking for a center. 

Of Walkways and Fountains

A Story

Once upon a time there was a business person who knew exactly what she wanted. So, she explained to an analyst precisely what it was that she wanted and all of the points that she wanted addressed and precisely how she wanted it addressed.  The analyst said he understood exactly what she wanted. 

So, the analyst went and assembled all the requirements and looked at everything that was spelled out. He gathered everything together. 

He found that he had some very usable information and some that was less than useable. So, he dug and dug and found the perfect item that would fit the needs the user described and make it pleasing to her.   

Then he assembled all the components and tested the product and found that it matched exactly what the user had asked for - and everything worked perfectly. The finished product was, indeed, a thing of beauty.

So, he called the user over to see the wonderful product he had made. She looked at it and said, "What is this?"

"Its what you asked for! It has everything you wanted!"

"No, this is..."

Have you ever heard of a project that matched the requirements precisely for what was needed to be included in the "finished product" only to find there was a complete mis-understanding about what the real purpose was?

Cast Curtain Call - Part 2 - Conversations

I was very fortunate this last week to have had extended conversations with several people, some "movers and shakers" and some "well respected testers" and some "regular folks." Rather than sort out which is which, I'm going to focus on some of the great conversations I had, starting Sunday evening, through Thursday.

The hard part is picking out the bestest ones. So, I'm going to summarize some and the mental meanderings that resulted.

Monday, chatting with Griffin Jones, he asked bout the mission and charter for the group I work with. We had been talking about techniques and some of the on-line forum conversations around exploratory/ad-hoc/fully-scripted testing in light of Michael Bolton's blog entry on Testers: Get Out of the QA Business. He asked about this after what I thought was a tangent that was around the question of "what works best for what kind of environment?"

His simple question got me to wondering, other than the slogan on the company's internal wiki about the QA/Testing group, what is it that we are about? For some time, we have been working toward getting more people involved in the idea of tangible requirements, of QA helping define requirements and acting as a bridge in the design process. But that begged the question - What is our mission?

I wonder how many testing groups (or whatever each group calls itself) have a "slogan" but no "mission" or "purpose" statement that can be pointed to, where everyone knows about it. If you don't know about it, is it reasonable for people to act towards that - its a goal, right? How do you achieve a goal if you don't know what it is (I feel another blog post coming on, but not right now!)

I had several brilliant little chats with Scott Barber. It helps when you're sitting next to each other at the Registration table. We talked about a bunch of stuff - For those who have read his stuff or have read his articles or postings in various online forums for that matter, he really is as smart as he seems - Holy Cow!

We got onto the "mission" of testing groups and "doing things right" vs "doing things well enough." What most theory-centric folks sometimes forget is that there is a cost to "doing things 'right.'" If the product will be shipped for 2 weeks late because you want to run a 4 week duration system load test, costing approximately $1M, what will the company gain? What are the risks? If you're extremely likely to see significant problems within the first 8 to 12 hours and the likelihood decreases over time, what will that extra two or even three weeks going to get you - other than a delay to delivery and a dissatisfied customer? That, in itself, is one reason why testers should inform and advise but not make the final go/no-go decision.

Yeah - there's another blog post on that in detail.

Other people I met included Jeff Fry, where DOES he get all that energy? Then Selenia Delesie was holding court on lightning talks in the lobby. WHOA! Crazy-smart and the nice as the day is long. Selena gave two really good presentations - unfortunately, while I read the abstract and supporting paper, there were not enough of me to get to all the presentations that I wanted to get to. I think that's a sign of a fantastic conference - too many good simultaneous presentations.

Other folks I met included Michael Hunter, the Braidy Tester - What a guy, although he's now braidless. Paul Kam from DornerWorks is another really smart guy. DornerWorks was one of the sponsors of the conference. They did a lot to make this happen.

Tuesday night the "Rebel Alliance / CASTAway Social" was a hoot. Tester games and chicken-wings and varied and sundry edibles and drinkables - Thanks to Matt Heusser for making that happen. He's another one who is just crazy-smart and really friendly. If you have not seen his TWIST podcasts, check them out.

After the social, a bunch of folks went to dinner and had a fantastic time. If I recall correctly, there were 15 or 16 of us. I scored a major triumph by having Michael Bolton sit across from me at the end of the table. What an amazing time. Melisa Bugai was sitting with us as we discussed the likely causes of why the lights on the deck of the restaurant kept going out. Yes, we tested the theory when Melissa unplugged the rope light going around the railing. They all stayed on after that. WHOO-HOO!

The conversation, as any conversation with Michael, took many twists and turns. We talked on language and literacy and music and education and mental discipline and CBC radio shows and how each touched on testing. What a mind-bendingly enjoyable night.

Wednesday I had the great pleasure of dining with Lynn Mckee and Nancy Kelln - and my boss. Best part is, the boss picked up the tab! WHEEEEEEEEEEEEE! Another night of fantastic conversation on testing and wine and great food. Did I mention we talked about testing?

There were so many other great conversations - How can I give a recap of all of them? As it is, there is much to think on.