Monday, January 13, 2014

Pandora's Box Testing

Anyone who has been around software testing for more than a wee, tiny bit is likely to have heard such terms such "Black Box,", "White Box" (or Glass Box) or "Grey Box" testing.  There is one Box of testing that includes all of these approaches.  It is, I believe, the most prevalent form of testing in use, world wide.  It is also the type of testing that is talked about the least.  I believe this is because most people who practice this form of testing are either in complete denial that they are doing this or have no possible clue that they are doing this.

To Review -

Black Box Software Testing is testing done without, or ignoring, knowledge of the design or code. 

White (or Glass) Box Software Testing is testing done while making use of knowledge of design or code.

Grey Box Software Testing is testing done with some knowledge of the design or code. 

Pretty straight forward, yes?

Certain "experts" tell us that these things can give us ALL the testing types there are.  Testing is encompassed by these terms.  Of course, that presumes that we can test all the requirements and all the possible combinations and paths through the code and all the potential cycles and ... yes.  I think you see.

These, however, are mere shadows of the trust that most testers put into the test method they don't even realize they are using.  This is the most commonly used  test method world wide - bar none.
If we look at the sheer numbers of people using it then, by most commonly used measures, this counts as the overwhelming favorite.

If we accept that the numbers of people using the method measures something important, as certain advocates of certain commercial training programs would have us accept, then this is, hands down, the most accepted method ever. 

Pandora's Box Testing

Yes.  There it is.  The most commonly used  test method world wide - bar none.  I bet some of you reading this have used it and did not even know it.  I know that, in retrospect, I have used it as well.  The challenge is to not use it.

Let us consider why using this most commonly used method a bad idea. 

Pandora's Box, from Greek mythology, focuses on Zeus "getting even" with Prometheus the Titan, who gave fire to mortals.  So, Epimetheus, the brother of Prometheus, is given a gift - Pandora to be his wife.  Pandora is given a gift by Zeus, a container that is sealed.  Zeus tells her to never open it, and gives the key to Epimetheus with the same instructions.

While Epimetheus is sleeping, Pandora's curiosity gets the better of her and she takes the key from him.  Then, she opens the container.

Evil things flew out of the container - causing Pandora to scream and try to close it.  She fails to stop the evil from entering the world.  It spread around the world, to every corner.  Epimetheus rushes to her, having been awoken from sleep by her scream.  He sees what she has done.  They open the container and the last thing contained in it flies out - Hope.  In some versions, the Spirit of Hope, Elpis.

In most versions told to children, this is seen as a positive sign.  A good thing to counter all the evil that had been loosed in the world. 

This is an incorrect interpretation. 

Hope, without positive action to support it and make that which is hoped for come to fruition, is the greatest evil of all. 

And that is what most "testing" that is done, world wide, actually is - trusting in hope.

When "testing" measures "quality" and "quality" is defined as conformance to requirements, are we acting based on reason?  Can we be certain that the requirements are right?  Can we test positive and negative conditions for the requirements? Can we only test the requirements?  Is there anything else we can test that would provide information about the software?

Do we have an interest in doing more than this?  Do we have a reason to do more than this? 

Are we trusting to luck in our testing if we don't do more?

Are we hoping that is good enough testing?  


  1. It drives me nuts when I hear people still using those terms! Just as it drives me nuts when people say things like "testers should only do black box testing, otherwise they'll be influenced by those (evil) developers (trying to sneak bad code into production, I suppose?)

    1. Heavens - Don't let it drive you nuts. The poor people are simply hoping they understand enough to test the software.

  2. "Pandora's Box Testing" - nice!

  3. I like the tembre of your post, but I still don't know what Pandora's Box testing is. You need one more paragraph, I think. Sounds like you are talking about unskilled testing based on fairy tale ideas about requirements.

    The distinction between black box and white box testing (I think the obvious term is "open box") is slightly interesting, but the terms don't represent heuristics of testing so much as they are descriptions of an incidental point of view.

    1. Thanks for the comment!

      "... you are talking about unskilled testing based on fairy tale ideas about requirements."
      Yes. Precisely.

      "black box and white box testing (I think the obvious term is "open box") ... terms don't represent heuristics of testing"
      I understand what you are saying. Alas, the people who focus on Black/White Box (I like "open" I may steal that) Testing find the idea of testing heuristics to not fit their model - their world view. Much of what they describe as "testing" I consider "checking." Based on my understanding of your work, I believe you might use the same term.

      The result is a very spotty effort to evaluate software. When people I have encountered/worked with have that view, they aggressively avoid "knowing the design" or "looking at the code" because it would corrupt their work. I strongly disagree with this.

      These tend to be of the same ilk who state they can not do testing because they have no requirements. Which tells me they understand neither.