On Testing and Quality Engineering

The other day I read an article on how Quality Engineering was something beyond testing. It struck me that, in the course of reading that article, it struck me that the author had a totally different understanding of those two terms.

Here then, is my response...



On Testing and Quality Engineering

A common view of testing, perhaps what some consider is the "real" or "correct" view, is that testing validates behavior. Tests "pass" or "fail" based on expectations and the point of testing is to confirm those expectations.

The challenge of introducing the concept of “Quality” with this conception of testing brings in other problems. It seems the question of "Quality" is often tied to a "voice of authority.” For some people that "authority" is the near-legendary Jerry Weinberg: "Quality is value to some person." For others the “authority” is Joseph Juran: "fitness for use."

How do we know about the software we are working on? What is it that gives us the touch points to be able to measure this?

There are the classic measures used by advocates of testing as validation or pass/fail: 

·         percentage of code coverage;

·         proportion of function coverage;

·         percentage of automated Vs. manual tests;

·         number of test cases run;

·         number of passing test cases;

·         number of failing test cases;

·         number of bugs found or fixed.

For some organizations, these may shed some light on testing or on the perceived progress of testing. But they speak nothing about the software itself or the quality of the software being tested, in-spite of the claims made by some people.

One response, a common one, is that the question of the “quality of the software” is not a concern of “testing,” that it is a concern for “quality engineering.” Thus, testing is independent of the concerns of overall quality.

My view of this? 

Hogwash.

Rubbish. 

 When people ask me what testing is, my working definition is:

Software testing is a systematic evaluation of the behavior of a piece of software,
based on some model.

By using models that are relevant to the project, epic or story, we can select appropriate methods and techniques in place of relying on organizational comfort-zones. If one model we use is “conformance to documented requirements” we exercise the software one way. If we are interested in aspects of performance or load capacity, we’ll exercise the software in another way.

There is no rule limiting a tester to using a single model. Most software projects will need multiple models to be considered in testing. There are some concepts that are important in this working.

What does this mean?

Good testing takes disciplined, thoughtful work. Following precisely the steps that were given is not testing, it is following a script. Testing takes consideration beyond the simple, straightforward path.

As for the idea of “documented requirements,” they serve as information points, possibly starting points for meaningful testing.

Good testing requires communication. Real communication is not documents being emailed back and forth. Communication is bi-directional. It is not a lecture or a monologue. Good testing requires conversation to help make sure all parties are in alignment.

Good testing looks at the reason behind the project, the change that is intended to be seen. Good testing looks to understand the impact within the system to the system itself and to the people using the software.

Good testing looks at these reasons and purposes for the changes and compared them to the team and company purpose and values.  Are they in alignment with the mission, purpose and core values of the organization? Good testing includes a willingness to report variances in these fundamental considerations beyond requirements and code.

Good testing can exercise the design before a single line of code is written. Good testing can help search out implied or undocumented requirements to catch variances before design is finalized.

Good testing can help product owners, designers and developers in demonstrating the impact of changes on people who will be working with the software. Good testing can help build consensus within the team as to the very behavior of the software.

Good testing can navigate between function level testing to broader aspects of testing, by following multiple roles within the application and evaluating what people using or impacted by the change will experience.

Good testing can help bring the voice of the customer, internal and external, to the conversation when nothing or no one else does.

Good testing does not assure anything. Good testing challenges assurances. It investigates possibilities and asks questions about what is discovered.

Good testing challenges assumptions and presumptions. It looks for ways in which those assumptions and presumptions are not valid or are not appropriate in the project being worked on.

Good testing serves the stakeholders of the project by being in service to them.

What some people describe as “quality engineering” is, in my experience, part of good software testing.

 

On Motivation, part 2

As the discussion I was having with the Unicorn at the coffee shop was winding up, a fellow I worked with a few years ago came in looking rather, frazzled. He joined us, although he looked rather askance at the unicorn. We made small talk for a bit. He had been promoted some 6 months before to a manager position and seemed frustrated.

The reason he seemed frustrated eventually seeped out. He was frustrated. He was trying to get "his resources" to "engage" in some new methods of doing things.

About this time, the unicorn bowed out and excused himself. I'm not sure this fellow even noticed him sitting at the table.

When we had worked together, he struck me as one who was perpetually looking to make a mark in some way. He always acted as if he knew better than anyone on the team or in a discussion on addressing any problem. He made sure that he offered advice to team leads and managers on how to address a problem - which normally involved wholesale changes to bring whatever was under discussion to be brought in line with whatever his set of beliefs were at the moment.

Funny though - his "beliefs" tended to shift. I'm not sure why.

It almost was as if he looked at whatever the situation was - and decided it needed to be different. Why things were the way they were or how they got that way did not seem to matter.

He deemed them valueless and needed to be completely replaced.

I got pretty tired of it after a while. When I was moved to a different group following a reorganization (yeah, these guys did that every 6 months or so) I did not miss the turmoil or drama of someone ranting about how screwed up things were.

Back to the coffee shop...

So, the fellow was trying to get "his resources" to "engage" in new methods of doing things. The challenge was that people were pushing back. They had always grumbled. Now, they were refusing "to cooperate."

And he was frustrated.

So I took a deep breath and tilted my head, just so, and asked "The processes that were in place before, the ones you replaced. Why were they implemented?"

I think he wanted to glare at me. Actually, I suspect he wanted to punch me. Instead, he said, "Look. This is stupid. I know what needs to be done and how things should be. And they just don't want to do it."

And I sipped my coffee and asked, "Remember when we used to complain about the 'policy du jour' and every 6 months everything changed, unless a new manager rolled in sooner than that? Remember how we used to kvetch about things changing for no apparent reason?"

He glared at me. Frankly, I think he thought I wanted to hit me. (That is funny to people who have met both of us.) "Look," he says, "the problem is these people just don't want to embrace anything new. It is not me or my problem - it is them."

He left the coffee shop. I suspect it may be a while before he goes to that coffee shop again.

The Problem

I suspect that is a pretty good summation of the view of people - managers, directors, VPs, dictators, whatever - "It is not me, it is them."

The irony is, in my experience, the first and foremost rule of anyone looking to change or improve things is - Learn and Understand how things got the way they are.

It is rarely as straight-forward as some would have it. Problems exist - Processes exist - Processes are normally introduced to address specific problems. Other problems may not be addressed by the changes, but, these are usually judged to be lower priority than the ones that are being addressed.

So, new Managers, Leads, VPs, Directors, Bosses... whatever - Before you make changes, I have found it to be a really good idea to take the time in how the organization got there. Even if you "watched" the "mistakes" happen - it is unlikely you were in the discussions that looked at the needs, the problems and the alternatives that got you to where you are.

Motivation?

If you want your "resources" to "get on the bus" and support you, I suggest you take the time to learn these things. Without doing so, it is almost certain that the people you expect to do the things you are mandating, will give your direction and instructions the appropriate level of effort and dedication.

None at all.

Because, when you move on, all these changes will be changed, and nothing will really change.

So, what is the motivation you have to make changes? Are you trying to "make your mark?" Or are you trying to do what is right for the organization?

On Motivation, part 1

I recently wandered into a neighborhood coffee shop for a little defocusing - and some of their Kenyan roast coffee and a fresh scone. While in line to place my order, my friend the unicorn walked in.

We had not intended to meet, it was just a happy chance. We sat down with our respective coffee and began talking. As happens sometimes, the 'catching up' developed into talking about something of interest. In this case, we found ourselves talking about motivation. We quickly set aside the stuff about "motivating people" and turned to forms of motivation - what motivates, maybe inspires, people to do work.

Most technical people we know who seek advancement and promotion into leadership or management positions fall into a few groups. Now, this isn't a terribly scientific study, just what the unicorn and I have seen.

There are the folks who really don't want to manage people and like having their hands dirty - they like the technical challenges that come with bigger titles and pay-grades.

Then there is the other major group - They want to lead beyond a technical perspective. They want to be "in charge"

The first type - These are the same type you find in very technical enlisted roles in the military - they soar through ranks at lightning speed. They display astounding prowess at tasks that others cannot comprehend. They show others how to do things, then dive in next to them in the doing - teaching their juniors what they are doing, how and why. They leave officers shaking their heads at how astoundingly well they do their jobs.

Until they get to the level where they "supervise" others. Then they don't get to do what they really like doing. Then they watch other people do what they want to be doing. And the longer they are in, the higher the rank they achieve and the further they get from doing what they truly want to do. So they leave - they don't reenlist.

In Corporate-Land, these same people, if they get assigned or promoted beyond "getting their hands dirty" and doing what they like doing, tend to resign and take another job. 

The second type - These are the folks who want to get into "leadership" positions. They are the movers and shakers and the up-and-comers in the organization.

Some folks have a negative view of everyone who is in this second, broad group. Neither the unicorn nor I can really fault people for having ambitions or desires. Nor could we really find fault with people wanting to get ahead and move up the ladder.

After all, if they are reasonably competent in technical roles, maybe - just maybe - they will remember what it was like in those roles as they move up in the organization chart.

For me, when dealing with managers or directors or other boss-types, I find it helpful if they have some appreciation of the challenges of the work done by technical folks, be it developers, DBAs, testers, whatever. While they may not be able to help from a technical perspective, they may be able to offer assistance in other ways, for example, running interference with other, less technical managers or functionaries.

People growing into roles that challenge them is an excellent thing. It is a desirable thing in my mind. Granted, the roles I have moved into have not been management ones. My forays into management have convinced me that I do not have the right "makeup" for managing others.

I salute those who do have that makeup and make full use of it. Indeed, I salute those managers who are motivated to manage others well, and help those they manage discover what it is that motivates them.

A third type - These folks who want to get into "leadership" positions for reasons I find to be less than honorable. Maybe you have heard that "Power Corrupts." I find the question of why one seeks power to perhaps shine a light on just how true that is, or is not.

Some people have something less than altruistic motives. Some desire high rank for achieving their own ends - their own self-aggrandizement. In these instances, I suspect the corruption has already occurred - and the quest for power is, in fact, the motivation.

The unicorn blinked at me.

He said something to the effect that people have their own motivations. He chuckled (a scary sound, frankly) at the thought that some of these sounded like Death Eaters. I did stop a moment and consider.

I was reminded that individual people are motivated by different things and these generally are internal to each of them. Their motivation drives their choices and how they work, just as mine do.

I can accept or reject those motivations and actions based on my values and what I hold important.  I can also choose to not associate with those whom I find I can not support.

On Service, Servants and Software

A few weeks ago, I was having a glass of wine with a couple of colleagues one evening discussing the role of Software Testers in developing good and well-performing software. There were some of the oft-stated lines about "not running the project" and "not owning quality" and "you can't test quality in" and so forth. We dismissed them as trite and irrelevant.

Where we landed was:

Software Testers serve the needs of the project and support other participants and stakeholders.

This brought up ideas around how the above can be interpreted. One person made an observation of a distinction between "service and servants" - one apparently triggered by something he heard or read and could not recall the source.

This sent us into the question of what "service" meant. Being who we were, we wandered off to distant times to discuss this idea. Of old, the Samurai of Japan were in service to others - at least in theory. In Europe, knights held their position through service to their lords. In both cases, it was possible to lose one's station - that bit often gets left out of the romantic stories. The romantic stories tend to overlook some parts and emphasize others. Reality was never as neat and tidy as stories, books and movies would have it.

Still, we looked at the idea of serving others.

Testers are in service to others. 

Then again, software developers are also in service to others. As are others in the various roles around software development. As are those whom we develop software for. The needs being addressed usually are problems that need to be addressed to support those whom they, in turn, serve.

People serve people who serve people.

Most folks, in their working life, serve someone else. We don't want to admit it, but unless we who are paying others - who serves whom? How do we get the money we earn?

The simple fact is, being in service and serving others are closely linked. I find people who object to being "servants" are people I cannot completely understand. When they object to being "servants" I look to see how they treat those who serve them.

It dawned on me in that discussion that there was much some people could learn about service, servants and servitude from people who are in service. Most Americans don't really know what that means or entails, to be "in service."

Perhaps that is part of the problem.

Some people see people who serve others as some form of lower life than they are. They have adopted a Victorian or Edwardian view of "station in life" - maybe they watched too many episodes of "Upstairs/Downstairs" or "Downton Abbey."

They see the films or shows where the household staff (servants) turn and face the wall when the family in the house, the ones they are "in service" to, pass them in the hall or stairway.

So now these folks treat waitstaff at restaurants as inferiors. They also tend to look down on hotel staff, flight attendants, sales clerks, construction workers, the simple minded, physically (and mentally or emotionally) handicapped, emotionally damaged, traffic cops, TSA agents, teachers, administrative assistants, clerical staff, med techs, nursing assistants, gardeners, Mexicans, Asians, Indians, or any other they see as beneath them.

I suspect, when one has such a superior opinion of themselves and a low opinion on lesser beings, that it is easy to look down on others - that the thought of being looked down on by others is repugnant.

If we, as testers, serve others, does that make us lesser beings?

Does that make us inferior?

Hardly - unless your ego is so fragile that it can't handle the simple idea stated above.

I've been in software for longer than some folks with such attitudes. I know as a developer, business analyst, project manager or software tester, my role exists so I can be of service to... someone else.

As a person in software development, I am a servant to a broader purpose. My purpose is to aid the project, make the software better, and by extension, make the company better.

Yes, Testers provide a service.

We are in service.

We are servants.

We serve for the betterment of our organizations, our craft and ourselves.

We are second to none.

How Advice to Beginning Pipe Bands also Works for Testers

When I was in demand to teach drumming workshops for pipe bands, I used to wrap up weekend sessions with new, or new-ish, bands with a session for both pipers and drummers. Usually, after they have had a practice together.

The typical format for these workshops was generally straight forward. I'd roll in Saturday morning between 7:30 and 8:00 AM with coffee and a pile of donuts and bagels (something like at "Take 10" from Tim Hortons, where an entire pot is in a take-away box with a stack of cups and sugar, milk, cream, etc.,) and a mix of donuts. People would start showing up a bit later, find coffee ready - at least enough to hold everyone over until a pot was brewed and ready at the venue. I'd set up the materials needed for the day while folks were chatting and drinking coffee, although some opted for power/energy drinks they brought themselves.

We'd work in sections, according to a pre-defined schedule. For example, beginning snare drummers at 9:00, more advanced snare drummers around 10:30. Break for lunch around 12 or 12:30 and back at it again as soon as possible after that. When I finished with each group, they'd head off and work on their assignments/exercises in a corner or a different room. Early afternoon, I'd work with tenor and bass drummers, typically, around 1:30 or 2:00. Around 3:00 everyone would come together and play exercises as a group - and then work on playing music.

We'd break around 5:30 or 6:00 for supper. After that, while officially "done for the day," usually, I'd work with people one or two at a time - if they wanted to. That would be more informal and very relaxed as we were all pretty tired by then.

Sunday, we'd start a little later, depending on venue and the individual band. We'd spend a lot more time working as a full group, working on the music and exercises from the day before. If someone was having a challenge with a passage, I'd spend a little time with them. If they still needed help, I'd send them off with someone who had it right to work on their own. The emphasis on Sunday was to work as a group, together, to make sure everyone was progressing - and to be ready to play with the pipers when they joined the drummers.

Usually the results were pretty good. The drummers were pleased to have made obvious progress. The pipers were pleased to hear the drummers playing with them and sounding good. At the end of the practice, I'd give a wrap up talk and encourage the drummers in particular and the band in general to keep working.

A typical message was something like:

The band has made huge strides. You all have come a very long way from (some point earlier.) This weekend the drummers have worked their butts off trying to get things just right. There has been really good progress, and there is still more work to do.

At the level of band's development and performance, there are a couple of things to think about and a bunch of stuff to set aside and ignore.  Don't worry about wearing the latest style of uniform. If the band can afford only the basics - kilt, shirt, cap - then so be it. If the kilts don't match - so be it. You are starting out. All that stuff will come. Don't worry about designing band cap badges or patches for the shirts or who wears what insignia. Don't worry if the drums don't match in make or color.

All of that will come in time. All of that will come as you get established and play out in public.

Don't worry about anything anyone else says to or about you. Don't worry about the condescending comments that some pipers or pipe bands will make. Don't worry about what other bands have or what attention they get.

Don't worry about seeking the same attention they get.

Work for yourselves. Practice hard and well. Practice individually. Practice with small groups. Rehearse as a band. Play well with the best sound, tone and execution you can achieve.

That will get the attention of people you want to get the attention of. If you are competing, the judges will know you are a new band and will not worry about details like if the kilts don't match. They will be focusing on what you do as a group. They will listen to how you perform. Perform well and the rest does not matter.

Let others worry about the periphery. You worry, and do something about, your task at hand. Let your fingers and hands do your talking for you.

To testers, I'd like to say something similar.

The people shouting for attention and making a big deal about what they are doing - let them. Truth will out, as is sometimes said. Eventually people will figure out if there is something to their shouting or not.

Don't worry about them.

Don't worry about what they say. Really don't worry about what they say about you.

Let them.

Do what needs to be done to support your organization.

Educate over time by doing good work, then explaining the work you do - gently, in language that is understood by the people asking. Work to develop your understanding - and share the understanding you develop.

Share that with developers you work with, other testers, people who ask at the coffee or snack station or cafeteria.

You don't need to be a name at conferences or on twitter to be able to influence other people and help them learn.

Most people doing good work are doing it quietly, getting the job done and moving on to the next item.

The clanging gongs of people shouting about how cool they are are just that. Noise.

Let the work you do, do the shouting for you. 

Hanging With the Cool Kids and Self-worth

I don't remember what grade I was in while at school that I realized I was not one of the cool kids. I was not one of the guys good at basketball, I was OK at football, but there were loads of folks better. I was slow, had terrible hand-eye coordination (can't be that astigmatism that has been addressed partly with my cataract surgery a few years ago...) - and as sporting ability was one of the defining points for cool-kids at that school, that was a significant handicap.

Then there was "social standing." My parents were nominally "middle class." At one time, they might have been considered "artisans" but to some folks, how much money one makes determines significance in society.

Alas, that seems to have not changed very much in the intervening years.

How much money one makes determines how important one is.

When I was in elementary and high school, that translated into the material goods the kids in school had - the shoes, designer labels, the other sundries students needed or wanted.

I expect that has not changed much either.

Since I was not very good at most sports, and my parents did not have loads of money to throw around - well, that was two strikes against me. So, I carried on as best as I could. I expect most people do.

Some folks made a point of rejecting the cool kids. Looking back, that seems a defense mechanism. "They did not reject me, I rejected them." I've seen it my kids and grandkids. I expect I did something similar.

So, on to college/university and hung with people who worried more about what we did and learned than what our parents did or how popular (and why) we were or who wanted to be seen with us and who we  wanted to be seen with. I guess, the folks I hung with then had an influence on me now.

and now...

I'm not a cool kid when it comes to testing. At one time, I thought "Gee, since testers are not among the cool kids in software, we must be all pretty decent folks."

Then I learned people did not really all think that way. There were folks who had significant ideas what testing was and was not. They were convinced they were right. Those who agreed with them were right also, and were "cool."

Except the folks who disagreed with them were also convinced they were right. The people who aligned with them were "cool."

Except there were levels of "cool" within both groups. There were the ones who leveled everything on a couple of issues or points. Maybe a single issue. If you agreed with them on that issue, you might not be cool, but you were not "misguided" or "wasting their time."

At least, not completely.

They may not let you into their "inner circle" but they'd talk to you at conferences. Maybe. Kind of like the cool kids who knew your name in school and sometimes spoke to you in the hallway.

It became clear to me some time ago that I am not a cool kid among testers.

I thought I wanted to be. I have learned clearly, that I am not. There are people who want to be included with the cool kids, one group or another. There are people who, like the hangers-on at school, want to be included and gleefully repeat the words of the cool-kids and retweet them and, and, and...

And so many people turn off their brains when the cool-kids speak. They so want to be accepted that - yeah - they are good with that. No matter what it is or means. They are fine with that.

Screw the cool kids.

Look to yourself. Look to your own journey. Find the things you think are important and work on those things. If some "major name" won't talk to you because you disagree? Screw them. They are not worth your trouble or effort.

Look to how you can improve the people around you. Help them get better and help yourself continue learning. Sure. The cool kids are all talking about some book. So what? Do you feel compelled to read it or quote it because they do, or because it makes sense and has a bearing on what you are trying to do?

Challenge your beliefs. Do you believe them because the cool kids said this is what you need to believe? Or maybe, you believe them because you have seen how things work in the wild?

Or maybe you reject some idea from the cool kids because you have not found they are true for you.

When you get to the point of thinking for yourself - you will realize that you can redefine what it takes to be a cool kid. Do what needs to be done. Learn what needs to be learned and share it with anyone who will listen, read, whatever.

Then you know what that means?

You are the cool kid.

On Facing Forward

The lady-wife and I were enjoying a glass of wine in the quiet evening last night. I was relaxing from a crazy-busy week at the day job. She was recovering from her own week of challenges. We both commented on the end of summer.

Mind you, we both can read a calendar and we know when summer "ends." We have had remarkably mild weather the last week or so. For August, it has been very cool. We have had a decent mix of rain and sun. The garden is absolutely loving it.

The squash are flourishing as are the tomatoes and the various plants in the salad garden. The potatoes are looking good as well. Still, the weather has been very cool - more late September than late August. I'm not minding too much, the air conditioner has not been running, so lower electric bills. The rain has been consistent enough where the rain-barrels have not really been empty since mid-July. So, lower water bills from less watering.

Looking Forward

We are looking forward to our annual excursion in a couple of weeks to the Wheatland Music Festival outside of Remus, Michigan. Absolutely massive weekend of music and friends and an excellent way to mark the end summer. We volunteer with the recycling crew, as we have for the last 20 years or so. The lady-wife talks with people about ideas for recycling and projects and simple things that people can do at home that save them money and are good for the environment. I do less of that and more of "immediate need" stuff - like dealing with compost-able materials from the various kitchens and food vendors. Which includes shovels and work gloves and improving the quality of the soil - in a couple of years.

We cold camp among pine trees with friends - play music and share food starting Thursday afternoon (we get up before the festival opens to help get stuff set-up) through Sunday afternoon. Coffee made on the camp-stove, prepared food we heat up and enjoy - along with various adult beverages being passed around and songs and tunes shared all make it a special weekend for us all.

It is something we look forward to all year.

When we first began volunteering, we did what we were told. We did what needed to be done. We shared laughs and, in my case, a fair amount of manual labor. We drank beer and coffee and had a good time. One year, we cooked bacon and eggs Sunday morning in the recycling tent as we had brought way more than was needed - and the food needed to be used. So, we shared.

There were "crew leaders" and "shift leaders" who made sure everything that needed to be done was done - when it needed to be done. Now, 20 years on, we are in those roles. Our "crew" this year includes the kids of people who were our crew and shift leaders when we started. One year, our oldest grandson was part of the volunteer crew.

It is strange, to some folks I expect, that part of what we are doing when working at this Festival, is teaching people what we know about composting, recycling, teaching and learning. We are also teaching people to take over our jobs - to do the things we are doing at some point in the future. 

The work I do at this Festival, really is "young man's work." It is physically demanding - even when you do manual labor on a regular basis. I expect this might be one of the last years where I actually do that level of work, instead of scheduling others to do it - and coach and encourage and cajole.

We do that anyway - it comes with the territory. But, us old folks jumping in and working as hard as the youngsters helps set an example to them, and to the casual observers walking past. I rarely end up with my clothes being anything but filthy at the end of a shift. That may be coming to an end as well.

A lot of things are that way.

In a matter of weeks, I will move from "Member of the Board of Directors of the Association for Software Testing" to "Former Member..." I am in the middle of transitioning away from being the Conference Chair for CAST - and returning to a back-bencher.

Several people at this year's CAST mentioned, off-the-record of course, the energy of the conference. The observation so many people made ran something like "Wow. The energy at this year's conference was amazing. There were loads of people who were eager to talk and share ideas and it was fantastic."

That made me feel very good.

Others commented on something else - "Wow. There were some really fantastic speakers and not very many were 'big names.' There were a few present and the tutorials were fantastic and lots of good ideas were shared. And still, these were not the major names people look for at conferences and they had great information and ideas to share."

Bingo.

That folks, is part of handing things on to the next generation of testers.

The amazing thing to me is that, stepping down as one of the "experienced" members of the board - I am wrapping up 3 years of service. The members who were not up for re-election have all served one year. All are experienced testers. They have management experience - and they are taking up the reins to direct AST.

They have knowledge and ideas.  They have energy and drive.

I am excited for the future of AST.

Getting something into good shape, or the best shape you can get it, so it can be handed off to the next generation is part of what makes the world what it is. It takes people being willing to hand over the responsibility to the next generation. It takes people in the next generation being willing to take on and do the jobs that need to be done.

That is part of people, old guard and new, facing the future together.

Face Forward.

Eyes to the Front.