Monday, May 23, 2011

Where No One has Gone Before: Exploratory Testing Lessons From Jean Luc Picard

Great.  You started out with a plan, maybe you had scripts to follow, maybe you had a nice neat plan - and before you know it you're off in the weeds.

Or worse, before you know it you've stumbled into some area that is completely uncharted and unknown.  Its as if you were navigating a sailing ship 500 years ago and realized you were smack in the middle of the area that said "Here be Dragons" or some OTHER undesirable slogan. 

None of us ever intend to get that far "out there" - well, I don't normally anyway.  Most of the folks I've worked with don't normally get that far "out there" either.  Usually.  Unless we feel like - well - seeing what's out there.

When that happens you have a couple of choices.  You can punt and start over, writing this off as a weird anomaly.  Sometimes when this happens folks shrug and say something like "I don't know how this happened. It must have been something I did wrong."  Frankly, I've said that once in a while as well.  Sometimes when I do that, in the course of re-tracing what I did I find where I should have zigged and I actually zagged.  I sometimes will make a note of it to return and intentionally follow that path after finishing off what I intended to do. 

The odd thing is that sometimes I find myself out in the weeds again, just as unexpectedly as I was the first time.  So, the choice we all face is to see if we have an idea what caused the event this time OR we can see where we can get from where we are right now.

I sometimes think of this as "X-Treme Exploratory Testing."  Instead of blasting our way through whatever we just ran into, we sometimes need to carefully unravel the threads that we have around us.

Do you remember the Star Trek TNG episode where the Enterprise picked up their own distress call saw a massive explosion and a debris field that was their own ship?  It was kind of like Groundhog Day - they found themselves in a "time-space continuum anomaly" where they repeated the same incident without knowing it. 

As luck would have it, several of the recurring characters had a sense of deja-vu - around the time they picked up the distress call as I remember it.  Then Data began saying things that were, un-Data-like.  So, they decided to try and send messages to themselves each time the repeated the process to let them know what they had tried - and they'd know if it worked or not by them not blowing up.  Cool, no?

What if you don't have an Android - humanoid artificial life form, not the phone - to tell you what did not work?  What if you find yourself trying to repeat the same process and finding yourself back in the weeds everytime?

For me, the simplest tool is a legal pad with a pen - and I make note of what is done.  See?  Who needs Lieutenant Commander Data when I have Ensign Note Pad?  ;)  Now, since I began doing this all kind of other cool tools have become available - Rapid Reporter is one.  Try it - I've read rave reviews but have not had the opportunity to put it through its paces myself - I look forward to it in the near future, however.

My point, such as it is, Don't be afraid of the unknown.  We're TESTERS - that is what we do!  We find the unknown and make it known!   We head off into the weeds and chart a course out and back again. 

If you stick to the path laid down in the script, you will have a fairly safe round of exercises.  It won't be testing - but it will be safely predictable and you will be able to show nice charts and pictures showing what is done and what is left to be done, and you may even find some bugs. 

It is when you head out to see what you can see that you really learn the product, the application or how it works. 

It is your job to boldly go where no one has gone before.


  1. Nice post, Pete!

    I had to chime in because I'm also a fan of the yellow legal pad for testing notes. I actually stumbled recently on something I enjoy from Franklin Covey called the "Better Than a Yellow Pad: Tech Notepad."

    I've tried Rapid Reporter a couple of times only, and I was excited by its simplicity and handy output, but I wasn't able to find a groove with it right away. Eventually I'll give it more of a fair shake because I think it'd be more efficient than what I do, but there also seems to be something special for me about recording and thinking on paper -- maybe it's feeling less self-censorship when not being recorded on the computer; maybe it's thinking in a space physically separate from where the testing takes place; or, hey, maybe there's just an instinctive allure to yellow paper.

  2. Hi Pete, thank you for a great post. You really pinpoint a thing I have been thinking of recently
    "It is when you head out to see what you can see that you really learn the product, the application or how it works. "

    And this is so true! I have seen fellow testers that are highly following the scripted tests, and while they are at it create new ones like nothing else. Really good if /anyone else/ would be next up testing, but noone wont do that. And at the same time, they themselves are not learning about the product or how to use it in a proper way. Even worse when thinking about the underlying logic and flows under the surface, the understanding is zero.

  3. Thank you both so much for commenting. I find it interesting when I run across "testers" who only follow a script and see no reason to vary from it one iota. It makes me wonder what happened to kill (or at least limit) their curiosity.