On Testing and Quality Engineering

The other day I read an article on how Quality Engineering was something beyond testing. It struck me that, in the course of reading that article, it struck me that the author had a totally different understanding of those two terms.

Here then, is my response...



On Testing and Quality Engineering

A common view of testing, perhaps what some consider is the "real" or "correct" view, is that testing validates behavior. Tests "pass" or "fail" based on expectations and the point of testing is to confirm those expectations.

The challenge of introducing the concept of “Quality” with this conception of testing brings in other problems. It seems the question of "Quality" is often tied to a "voice of authority.” For some people that "authority" is the near-legendary Jerry Weinberg: "Quality is value to some person." For others the “authority” is Joseph Juran: "fitness for use."

How do we know about the software we are working on? What is it that gives us the touch points to be able to measure this?

There are the classic measures used by advocates of testing as validation or pass/fail: 

·         percentage of code coverage;

·         proportion of function coverage;

·         percentage of automated Vs. manual tests;

·         number of test cases run;

·         number of passing test cases;

·         number of failing test cases;

·         number of bugs found or fixed.

For some organizations, these may shed some light on testing or on the perceived progress of testing. But they speak nothing about the software itself or the quality of the software being tested, in-spite of the claims made by some people.

One response, a common one, is that the question of the “quality of the software” is not a concern of “testing,” that it is a concern for “quality engineering.” Thus, testing is independent of the concerns of overall quality.

My view of this? 

Hogwash.

Rubbish. 

 When people ask me what testing is, my working definition is:

Software testing is a systematic evaluation of the behavior of a piece of software,
based on some model.

By using models that are relevant to the project, epic or story, we can select appropriate methods and techniques in place of relying on organizational comfort-zones. If one model we use is “conformance to documented requirements” we exercise the software one way. If we are interested in aspects of performance or load capacity, we’ll exercise the software in another way.

There is no rule limiting a tester to using a single model. Most software projects will need multiple models to be considered in testing. There are some concepts that are important in this working.

What does this mean?

Good testing takes disciplined, thoughtful work. Following precisely the steps that were given is not testing, it is following a script. Testing takes consideration beyond the simple, straightforward path.

As for the idea of “documented requirements,” they serve as information points, possibly starting points for meaningful testing.

Good testing requires communication. Real communication is not documents being emailed back and forth. Communication is bi-directional. It is not a lecture or a monologue. Good testing requires conversation to help make sure all parties are in alignment.

Good testing looks at the reason behind the project, the change that is intended to be seen. Good testing looks to understand the impact within the system to the system itself and to the people using the software.

Good testing looks at these reasons and purposes for the changes and compared them to the team and company purpose and values.  Are they in alignment with the mission, purpose and core values of the organization? Good testing includes a willingness to report variances in these fundamental considerations beyond requirements and code.

Good testing can exercise the design before a single line of code is written. Good testing can help search out implied or undocumented requirements to catch variances before design is finalized.

Good testing can help product owners, designers and developers in demonstrating the impact of changes on people who will be working with the software. Good testing can help build consensus within the team as to the very behavior of the software.

Good testing can navigate between function level testing to broader aspects of testing, by following multiple roles within the application and evaluating what people using or impacted by the change will experience.

Good testing can help bring the voice of the customer, internal and external, to the conversation when nothing or no one else does.

Good testing does not assure anything. Good testing challenges assurances. It investigates possibilities and asks questions about what is discovered.

Good testing challenges assumptions and presumptions. It looks for ways in which those assumptions and presumptions are not valid or are not appropriate in the project being worked on.

Good testing serves the stakeholders of the project by being in service to them.

What some people describe as “quality engineering” is, in my experience, part of good software testing.