You Call That Testing? Really? What is the value in THAT?

The local tester meetup was earlier this week.  As there was no formal presentation planned it was an extended round table discussion with calamari and pasta and wine and cannoli and the odd coffee.

"What is this testing stuff anyway?"

That was the official topic.

The result was folks sitting around describing testing at companies where they worked or had worked.  This was everything from definitions to war-stories to a bit of conjecture.  I was taking notes and tried hard to not let my views dominate the conversation - mostly because I wanted to hear what the others had to say.

The definitions ranged from "Testing is a bi-weekly paycheck" (yes, that was tongue-in-cheek, I think) to more philosophical, " Testing is an attempt to identify and quantify risk."  I kinda like that one.

James Bach was also referred to with "Testing is an infinite process of comparing the invisible to the ambiguous in order to avoid the unthinkable happening to the anonymous."

What was interesting to me was how the focus of the discussion was experiential.  There were statements that "We only do really detailed, scripted testing.  I'm trying to get away from that, but the boss doesn't get it.  But, we do some 'exploratory' work to create the scripts.  I want to expand that but the boss says 'No.'" 

That led to an interesting branch in the discussion, prompted by a comment from the lady-wife who was listening in and having some pasta.

She asked "How do you change that?  How do you get people to see the value that you can bring the company so you are seen as an asset and not a liability or an expense?"

Yeah, that is kind of the question a lot of us are wrestling with.

How do you quantify quality?  Is what we do related to quality at all?  Really?

When we test we... 

We exercise software, based on some model.  We may not agree with the model, or charter or purpose or ... whatever.  There it is.  

If our stated mission is to "validate the explicit requirements have been implemented as described" then that is what we do, right?  

If our stated mission is to "evaluate the software product's suitability to the business purpose of the customer" then that is what we do, right?

When we exercise software to validate the requirements we received have been filled, have we done anything to exercise the suitability of purpose?  Well, maybe.  I suspect it depends on how far out of the lines we go.  

When we exercise software to evaluate the suitability to purpose, are we, by definition exercising the requirements?  Well, maybe.  My first question is, do we have any idea at all about how to judge the suitability of purpose?  At some shops, well, maybe - yes.  Others?  I think a fair number of people don't understand enough to understand that they don't understand.

So, the conversation swirled on around testing and good and bad points.

How do we do better testing?

I know reasonably few people who don't care about what kind of a job they do.  Most folks I know want to do the best work they can do.

The problem comes when we are following the instructions, mandate, orders, model, whatever, that we are told to follow, and defects are reported in production.  Sometimes by customers, sometimes by angry customers.  Sometimes by customers saying words like "withhold payment" or "cancel the contract" or "legal action" - that tends to get the attention of certain people.

Alas, sometimes it does not matter what we as testers say.  The customers can say scary words like that and get the attention of people who define the models us lowly testers work within.  Sometimes the result is we "get in trouble" for testing within the model we are told to test within.  Of course, when we go outside the model we may get in trouble for that as well.  Maybe that never happened to you?  Ah well.

Most people want to do good work - I kinda said that earlier.  We (at least I and many people I respect) want to do the absolute best we can.  We will make mistakes.  Bugs will get out into the wild.  Customers will report problems (or not and just grumble about them until they run into someone at the user conference and they compare notes - then watch the firestorm start!)

Part of the problem is many (most) businesses look at testing and testers as expenses.  Plain and simple.  It does not seem to matter if the testers are exercising software to be used internally or commercial software to be used by paying customers.  We are an expense in their minds.

If we do stuff they do not see as "needed" then testing "takes too long" and "costs too much."  What is the cost of testing?  What is the cost of NOT testing?

I don't know.  I need to think on that.  One of the companies I worked for, once upon a time, it was bankruptcy.  Other were less dramatic, but avoiding the national nightly news was adequate incentive for one organization I worked for.

One of the participants in the meeting compared testing to some form of insurance - you buy it, don't like paying the bill, but when something happens you are usually glad you did.  Of course, if nothing bad happens, then people wonder why they "spent so much" on something they "did not need."

I don't have an answer to that one.  I need to think on that, too.

So, when people know they have an issue - like a credibility gap or perceived value gap - how do you move forward?

I don't know that either - at least not for everyone.  No two shops I've been in have followed the same path to understanding, either.  Not the "All QA does is slow things down and get in the way" shop nor the "You guys are just going through the motions and not really doing anything" shop.  Nor any of the other groups I've worked with.

Making the Change

In each of these instances, it was nothing we as testers (or QA Engineers or QA Analysts or whatever) did to convince people we had value and what we did had value.  It was a Manager catching on that we were finding things their staff would not have found.  It was a Director realizing we were working with his business staff and learning from them while we were teaching them the ins and outs of the new system so they could test it adequately.  

They went to others and mentioned the work we were doing.  They SAW what was going on and realized it was helping them - The development bosses saw the work we did as, at its essence, making them and their teams look good.  The user's bosses realized we were training people and helping them get comfortable with the system so they could explain it to others, while we were learning about their jobs - which meant we could do better testing before they got their hands on it.

It was nothing we did, except our jobs - the day-in and day-out things that we did anyway - that got managers and directors and vice-presidents and all the other layers of bosses at the various companies - to see that we were onto something.

That something cost a lot of money in the short-term, to get going.  As time went on, they saw a change in the work going on - slowly.  They began talking about it and other residents of the mahogany row began talking about it.  Then word filtered down through the various channels that something good was going on.  

The people who refused to play along before began to wander in and "check it out" and "look around for themselves." Some looked for a way to turn it to their advantage - any small error or bug would be pounced on as "SEE!  They screwed up!"  Of course, before we came along, any small errors found in production would be swept under the rug as something pending a future enhancement (that never came, of course.)

We proved the value by doing what we did, and humbly, diplomatically going about our work.  In those shops that worked wonders.

And so...

We return then to the question above.  How do we change people's perspectives about what we do? 

Can we change entire industries?  Maybe.  But what do we mean by "industries?"  Can we at least get all the developers in the world to recognize we can add value and help them?  How about their bosses? 

How about we start with the people we all work with, and go from there?  I don't know how to do that in advance.  I hope someone can figure that out and help me understand.

I'll be waiting excitedly to hear back from you.