LIVE! at STARWEST! Day 2!

Thursday morning at StarWest is lovely.  Some clouds, comfortable temperature (for me) and we started out sitting drinking coffee and talking about testing.  What a great start to the day.

Heading off for some day-job work stuff, will be back for the morning keynote.

---

So, the work stuff took longer than I thought it would, so I'm a bit late to this keynote (which is disappointing to me as I wanted to hear it.)  Alas - 

Ben Simo is a really nice guy, mild and soft spoken.  Do not be mistaken into thinking he is anything but an extremely skilled  describing his experience trying to get health insurance coverage through HealthCare.gov as "The Power of an Individual Tester."

Summary - It ain't pretty.

You can see his thoughts, comments and learnings here:  http://www.questioningsoftware.com/ This is different from his regular/normal blog, here: http://blog.isthereaproblemhere.com/

The problems were Legion: poor security questions, poor account verification handling (as in "We sent you an email with your new password!" except it can take 3 days to get delivered and the link for the password expires in a couple of hours.

Then there were problems with how information was returned and displayed - anything from sensitive data and it personal information posted behind the screens, not visible - unless you trace what is actually being sent to the browser.

The "skilled hacker" comment came as a result of Ben noticing stuff and looking to see how far the thread went.  With a series of fairly simple steps, a person with bad intent could get everything they really needed simply by monitoring cookie streams, etc.

The data included in what was being sent to the front end included income and other sensitive data that should not really be returned to the browser in a secure system.  There were other issues around when and how data was collected from the screen, which is retained and passed back and forth.

Lesson: Technical people - be straight when talking with non-technical bosses.  Be absolutely certain that their understanding of how the system works is correct.

When there is bad news - be very precise.  Don't get involved in turf wars, give people information and be straight.  No matter how BAD the information is, get it out and then figure out how to deal with it.

Ben gives an excellent overview of problems in the site - starting with fundamental design flaws through transaction handling that had clearly not been considered.  The probability of success was low for such a complex system - then they made it worse by not making certain that public officials could give real, accurate information about the state of software.  Whether they chose to use it is beyond your control.  Be honest.

Ben does note that there are now some improvements - for example the actual ability to send feedback - and the form actually opens.

---

Next up – Rainmaking for Testers (and managers) with Julie Gardner.  She gave the opening keynote yesterday.  She’s with RedMind in the UK.

Julie talks about “trusted adviser” and “rainmakers” – and getting the image of what a “trusted adviser” has to say.  The issue is – what makes an adviser trusted?  How do people view you?  Are you the guy from Office Space?  How about a used car salesman?  How about Wormtongue from Lord of the Rings? 

Each of these were trusted, to some level, by some people.  They were not trusted by others.  This depended on several things, including the way that messages are delivered. 

It is important to be straightforward.  It is also important to frame bad news in a clear way – not overstating the problems, not understating the problems (note Ben’s keynote) – but give a concise evaluation.

Be willing to help, even if it is hard an outside of your normal realm, role and behavior. 

AND! She used one of my favorite words!  “Your manager does not want you to be a sycophant.” 

Don’t be a “yes man” a “suck up” – be straight with them.

(Pete Comment: OK – slippery ground here –)

What does “development” produce – CODE!

What does “testing” produce – (varied things shouted) Information – Documentation – Bugs

In general, too much detail will produce information overload, which will distract people from the message you want to deliver.  Be succinct.

Trust entails risk.  To be trusted is not a right – it takes time to develop trust over the questions at hand.  Speaking clearly and making sure that “the proof of the pudding is in the taste” – Make sure you deliver on the words you choose to use – the things you say mean things.  Make sure that when you say something – you can actually deliver.  People are taking on a risk by trusting others. 

What gets in the way of trust?  Bad manners – dishonesty – exaggeration – arrogance – empty promises – on and on and on…    Don’t do that…

Building strong relationships require you to ask questions and then listen to the answers – carefully.

Concentrate – listen (you have 2 ears, & 1 mouth – you also have 2 eyes – listen with your eyes as well. 

When you need help – ask.  You can’t be perfect for everything.  You can be empowering someone else by recognizing them.

Give advice effectively – Be prepared (as much as you can).  Advice is rarely a perfectly logical process – so be straight, don’t hide bad news and don’t forget the good news.  Don’t bury the bad news, be balanced when you can.

Know your audience – different people need different degrees of information.  Don’t give them the control panel for a jumbo jet, when the manager needs the information on the car dashboard. 

Now she’s talking about predictions- Number of tests you have to run, the number you expect to run by X time frame.  Then there is the question of predicting bugs… (Pete comment: can you really predict bugs? How?)  Possibly in some instances the information found in testing over time may help. (Julie refers to this as “defect measurement rate” which was developed by a colleague of hers.)

(Pete Comment – I know people have different views – I am not convinced that predictions of bugs to be detected is terribly helpful.)

OK – Moving on – Developing a helpful mindset is important.  Don’t place blame, don’t slag someone for not being perfect (you aren’t either.) 

Julie had a survey – “What do senior managers look for in a test manager?”

Top 10 common responses:

10 - Pragmatic
9 - Understand Testing
8 - Identify and Manage Risk
7 - Cooperation and Team Players
6 - Understand the issues and the politics around them
5 - Trend Analysis and forecasting
4 - Flexibility
3 - Abe to communicate at all levels
2 - Honesty & consistency
1 – Understand the Business

To move from “trusted adviser” to rainmaker – consider:
People do business with people because they choose to, not because they have to.  We can always find others doing the same thing or selling the same product.  It’s the personal connection that makes that happen.”

Even when marketing yourself – retain your integrity. Don’t “sell out.”

Testers can be the “Jiminy Cricket” for the team, project, organization, and act as the conscience.  We can be honest with people and give straight information – and ask if we are doing the right thing.  We must have the courage to do that – and speak truth, even when people may not want to hear it…

Adding the elements of “trusted advisor” with credibility, you get integrity.

Annnd – power is gone – it was a very good session…

---

Hallway (well, coffee line) conversation with a couple of people on testing, exploratory testing, applying ET in their environment and leadership.  Wonderful!

---

LUNCH!  Be back later with updates from James Christie's presentation...
Ummm- one thing - we don't have wifi in most of the session rooms - and limited power sources.  So, updates come as I can post them -
---

We’re BACK! 

After a lovely lunch table conversation with many people, including Griffon Jones, on Exploratory Testing, I’m now in James Christie’s session on “The Unfortunate Triumph of Process over Purpose.”  

James is best known, lately, for his opposition to the new ISO 29119 standard on software testing. I rather find this unfortunate – not his opposition, but that most people see him as a controversial figure.  I can say, after many conversations, James is an excellent thinker, writer and has very good ideas.  I have never heard him present and I am looking forward to this.

His talk is based on his career as a (process) auditor test manager, with examples from specific projects he worked on.  One example, the first, was a project done for a division of the British Government.  This project was politically sensitive at the time, and included significant expectations around accessibility (visually impaired in particular.)

In a sense, this was normal.  The only problem is that when you have a risk-averse organization the likelihood of failure tends to go up.  Instead of “Keep Calm and Carry On” the motto tended to be “Keep Calm and Ignore Reality.” 

ANNNNNNNNd… James refers to non-existent documentation – 70 pages of stuff that made no sense.  James role was to document and ensure that the items worked as intended.  Thus he made lovely documents, but much were of little value. 

Test plans were irrelevant because the documented requirements and the software developed had little to do with each other.  There was not a rejection of any form of Agile methodologies, they simply did things the way they always had.  (James is speaking really, really fast – banging points out – it is hard to keep up.)

The project was declared a success, even though they had just barely delivered something that sort of worked. 

By comparison, the crash of banks in 2008 resulted in several findings – the Parliament report on the failure of Royal Bank of Scotland had a dramatic point.  (See photo) It is an excellent summary of what happens when people worry about what the steps of the process are, and forget about what happens around the reason the processes exist. 

Thorstein Veblen described what he called “trained incapacity” (1914) where people are given such detailed, proscriptive methods that when the situation changes, they cannot adapt.  People are trained to not be flexible, and the result was people who were easily and often bypassed because they have been locked into one and only one way of working.

Isabel Meznzies Lyth (1959) described problems resulting from organization’s structure and processes: Task, Technology amd social/psychological needs of the people.  Lyth was looking at nurses, and how the growing view of them being fungible resources, thus nurses were seen as interchangeable. 

The problem was that these changes were not helping the people who needed to do the work, they were designed to help managers manage those people. (They also led to entrenched, fixed thinking similar to the Maginot Line built by the French in the 1930's against a German invasion.)

David Wastell demonstrated that structured methodologies did not actually advance the job, but they served as social defenses for the managers.  They showed no value in and of themselves.

James slides in to the ideas around “Rules versus Principles,” citing FD Roosevelt, “Rules are not necessarily sacred, principles are.” Principles are what we use as fixed points to hold ourselves – we use them to measure ourselves against our own values. 

Brenda Zimmerman describes the difference between complicated and complex.  Consider the space program, compared to raising children.  We can follow the same steps and get people to the moon and back.  We can follow the same steps with each child and, somehow, come up with completely different"results."

Cynefin – Welsh word meaning “whole man” from Dave Snowden.  This is a framework to help people make decisions and chose a response.   Hence, “complicated” situations are less prone to repeated processes (best practices) because, like raising children, it is difficult to master and predicting outcomes are pretty much impossible.

Snowden’s point is that there is no real clarity to what is needed. 

The center of the diagram is disorder.  This is where we normally are.  Most people will try and find a way to migrate to a situation that is easier to understand so we tend to impose those values from “Obvious”

The boundary between Chaotic and Obvious is a mess.  You can’t really balance between them  People who find themselves on that boundary are, as Snowden describes, in an “Oblivious” state – they think they are in Obvious, but are not – and will tumble into Chaotic. 

Most of the time, software projects are bouncing between Complex and Complicated.  This leads us from believing that “best practices” will save us – until we realize we are actually dealing with Chaos.

When you deliver software in spite of the process, you are subverting the process, not adhering to it.   

It is the crash into reality that ends the delusion.

James cites Wenberg’s “Second Law of Consulting” – “No matter how it looks at first, it’s always a people problem.”

(Pete note – this is around the 20^th time in the last 4 days that a different Weinberg reference was made – all different quotes – and just the ones I have heard, or made myself.  Thank you, Mr. Weinberg, for being so inspirational.)

“Any approach to testing that ignores “people problems” and tries to tame human nature with rules, standards and rigid processes is doomed”

Thus, any attempt to force people into a box will fail.  Full stop.

When you are so terrified of something, unless you focus on the people, you are almost certainly going to bring your worst nightmare true.

Questions – first one is around what does one do to respond the people who will say things that oppose all standards and processes. 

James’ response is that people don’t understand.  He also clarifies the difference between standards and conventions.  These are similar – but not the same.   (Pete comment: and this is a simple concept many people fail to realize.)

Well done, and well said, James. 

---

Right - after a brief break and more hallway conversations, I'm in the room for Paco Hope's keynote "Softwarts: Security Testing for Muggles" which he refers to as a course in "Defence Against the Dark Arts.Black Hats"  Paco has been walking about the conference off and on the last 2 days wearing... I think they are "sorcerer robes?"  Except given the title of the presentation, I'd expect they'd be more "Harry Potter" style than "Sorcerer's Apprentice" style.  Ah - well - its all good.

We begin with prizes from the test lab.

And Paco is off to a running start... well - maybe not running.  He starts by admitting that he is "mixing genres" which is kind of appropriate for security testing.  The thing is, it is hard to identify who the good guys and bad guys are - without some work.  They don't really wear black hoodies when they are doing their thing (to the best of our knowledge.)

Generally, security defects are not that different than other defects.  They CAN be different, but often times they land in the realm of unintended consequences.  "We meant to do X (which is good) except Y also happened (which is bad.)"

The thing is, just because the security defects appear to be different does not mean that they automatically are.

Similarly - there is a myth that "security testers" are inherently different.  They do stuff that no one else can understand, let alone do.  Its like Tom Cruise in Mission Impossible hanging in the middle of the room to get to the keyboard.  Except he's probably a "functional" tester that need to climb into this silly rig to trigger some branch of the code.

There are things functional testers deal with that security testers do not need to (like "code coverage".)  There is an idea that security guys have this mystique - this wand or something.

Except testers actually have test inputs & harnesses.  They have user stories, use cases and documented requirements (which serve as the magic map of the kingdom.)  And there are logs and profile info that can act to support you.

And he relates a story of how he found a hole in a system dealing with interest rates & large purchases (financial stuff.)  He found a hole in the system where he could change the rate (increase) in the confirmation message.  He thought this was a big deal - Apparently it was not to them (of course, they never checked to see if

Four Principles -
1 - Orcs not Elves
An Orc is a dumb brute with a primitive weapon and you can point them somewhere and something dies.  One orc is not a problem - 50,000 orcs could be.  Think - bot-nets which can be launched in denial of service attacks, or other malicious manners (and if you know where and how, you can rent time on one of these without actually needed to build it yourself.)

Offline attacks are powerful instances that allow an attacker to brute force a system - sure its hard.  BUT - it can be done.  Think, sniffers getting encrypted versions, decrypt them (brute force) and walk right in to the account.

Orcs can be applied to our own intents.  Like, write a program to create input daya; check it works, run an attack to see what is happening.  Can you withstand the demo attack you created?

2 - No Gold Required
You don't need to take your bag of gold pieces somewhere to buy magic devices.

There is stuff like OWASP with tools and resources available - free.  There is CVSS (common vulnerability scoring system) to help score security risks.  There is Kali Linux - which is pre-build and bootable.

These tools are pretty useful - people will help you - and it does not really need that much money to make happen (many are free, or can have help obtained for the cost of beer.)

You can get this stuff by doing a little bit of work.  Sometimes you can get information from various sources - like the experts themselves - via Twitter, mailing lists, things of that nature.

3 - Reverse Alchemy 
Instead of taking ordinary stuff and turn it into gold, we are going to take gold and turn it into ... crap.

HTTP proxies are powerful tools we can use - to do our thing - by watching what happens when information passes through it.  Pretty much everything runs through proxies - why not set up one of our own?

Secure connections? Well, if you're running your own proxy then you can see what is happening.  Cool, no?

This allows us to monitor, intercept then rewrite traffic in your own proxy.  You can capture stuff and tweak it.  The data you change is now what you want to exercise against, or through, your system. 

4 - Use a Spell Book
You don't need to memorize everything - have a spell book to help you recall the right command for fireball (or whatever.)

Keeping a reference handy for what you need to do - how you can make things happen by retaining links, setup information, whatever.  Spells -> Commands; Animating the Dead -> Simulations, and so forth.  The specific things you need to do are the contents of your spellbook.

Equivalence Class partitioning may relate to XSS or SQL injections - these may be "classes" of attacks or hacks.

Think about the things that "can't happen" or places where "this could not happen."  It is rare that systems have no vulnerability.  EG., a 3rd party service provide sending data can (and will) make mistakes.  That is a vulnerability.

Combine those ideas and voila!  You can begin to be a security wizard, too!

---

Some announcements - and I'm almost out of power.  more updates and pictures will be loaded.
And - pictures loaded - including this one of the ice cream that was available for the afternoon break on Thursday....  Yes, for those who have not been to the Disneyland Hotel, Mickey is impossible to escape or hide from (look at the pattern in the carpet.)

---

About to head to the airport.  I intend to post a summary/retrospective of my week from there.

LIVE! at STARWEST! Day 1

It is Wednesday morning and I'm sitting in Anaheim, California, at the Disneyland Hotel for the 2014 instance of StarWest.  Like many conferences, there are a couple days of training and tutorials before the conference itself.  Monday and Tuesday were the tutorial days for this conference.

Monday morning I spent working on day-job stuff, that afternoon I spent preparing for the presentation I am giving later today.  Tuesday morning, more day-job stuff in the morning, then met and sat in some conversations with testers around ideas about testing.  (Sorry, Paul, when I tried to get back into your tutorial, the nice lady at the door said I had the wrong code on my name tag.)

Wednesday started with a small but doughty lean-coffee sitting at tables outside - near the coffee shop & Goofy's Kitchen (yes, Goofy is here and stops by to say "Hello.")  A bit more day-job work to accomplish, then off to the keynotes!

---

Lee Copeland gave opening remarks for the day - and Julie Gardner is launching the first keynote of the conference.

Julie tweets at @cheekytester and is talking on "glueware" - the stuff that puts systems together and helps them stick.

Central to that is the idea of communication - and the difference between information and communication.  AND - my battery is going.  Summary later!

---
And BACK!  I have power AND a wifi connection!

I rather liked a fair amount of what Julie had to say.  I have issues with some specific points made, and those tend to be things I have issues with that many presenter say and write.  More on that later (maybe.)


In general, she covered the costs of distraction (note - Multi-tasking does not work.) She examined what happens when people face various constraints and limitations - and noted that people developing software to be integrated with  are operating under the same constraints as others are - conflicting incentives, unclear expectations, unclear approaches to development, uncertainty of use and customer needs and expectations.

These things give us customer/client responses like these:

As it was, I thought she gave a reasonable introduction to problems given the audience at the conference.  What do I mean?

In the last three days I have met many, many people attending a conference on software, testing or technology for the first time.  There have been many of them who are attending their first StarWest conference (as I am.)  There are many who are new to testing.

This is important to me.

People are here to learn, and in some cases, unlearn practices they have some exposure to and will be looking to advance what they do.

--------

Since I've been dealing with other issues, like finding power, I have no notes on the keynote by Rob Galen.  (I could hear him through the open doors, but missed too much to give a concise description.  I hope to address that as soon as I can.)

And now, I must go give my presentation - then lunch! :)
---

Right.  My presentation came and went – People were there and generally seemed to enjoy it.  Of course, no session is for everyone.  We’ll see what the reviews say.

Lunch was something interesting.  I volunteered to participate in a “Lunch with the Speakers” function – where speakers have lunch, while fielding questions from people sitting at the table with an interest in the topic of the table.  We had an excellent discussion that grew out of questions people had from my session.  I thoroughly enjoyed both.

Ran into Rob Sabourin in the hall after lunch.  I should note that I ended my presentation with “If you liked the session, I’m Pete Walen and this has been ‘Growing into Leadership.’  If you did not like it, my name is Rob Sabourin and this has been ‘Testing Lessons from Sesame Street.”

So I wandered back into the same room where I had presented to catch Martin Nilsson giving an excellent presentation on being a “social tester.”  He models this idea on the “types of testers” description James Bach did, some time ago.  He uses his career as an example of how people do things and did things in the past – like, walking around with a coffee cup talking with people. 

By reaching out and discussing things – asking questions about things OTHER than projects and problems – like, reading the Times of India when he needs to work with organizations with staff in or from India.  By being aware of the potential areas of concern for staff members and colleagues in the handling of not just information, but also of interactions, testers can change the dynamics of the project handling interpersonal connections. 

Martin then describes behaviors of teams, and again, using his experience, looking at how inserting himself into these teams that were effectively siloed, changed the complete dynamics in standups, etc.  The question around improving communication can sometimes be addressed by sharing a contact point – where information from developers and project managers and everyone else, simply became shared more openly.

This is an important point. 

If we consider Weinberg’s “All software problems are people problems,” we get a tool for understanding processes and project blocks. 

An example of informal models – Martin had meetings each week with test leads from different teams – informal meetings, like having a coffee in the morning together.  He then began inviting other people to join them – like, development managers, support managers, and others.   The result was that managers began going to the test leads for the project they were interested in, because they tended to have a better understanding of what was happening than project managers sometimes did.

This was an excellent insight for me.  I found that extremely valuable and a very good idea.

He mentioned, briefly, something around the idea of a person who kept cookies at his desk – anyone could have one, but he’d say “hello” and speak briefly with them, to get a better understanding of each of them.  This way, he had a bridge he could use when he needed information or was looking for insight into something he did not understand. 

Kind of like keeping chocolate at my desk.  (I expect the chocolate to be totally gone when I return from StarWest…)

----

BREAK TIME! 

---

After finishing the above update (there is no wifi access in the session rooms which makes live blogging for me a bit challenging) I visited the expo and had conversations with people I needed/wanted to chat with.  (Yes, folks at work, I'm bringing you stuff...)

I also had some interesting discussions with tool providers that I need to investigate further.

Now there is a reception and more networking stuff happening.  It has been a good day thus far - and the music is starting up in "Downtown Disney" - although it could be a sound check for the show later tonight...
---
Note: Most of my pictures of keynote presenters simply were not very good.  Sorry folks.  The people sitting in the middle of the room snapping images got much better pics than I did.

On Success and Negative Attention

I've been reading several instances lately where a person gets some measure of attention for work they've done or what they've written or said.  Then, someone else determines they are not deserving of the attention and should be called out on it.  They, in turn, find themselves being overrun by people more than happy to drag them down.

Alas, women are often the targets of these types of attacks.  That's what they are, after all, attacks.  It is nearly impossible to defend against them without drawing more.

Sadly, this is not a new phenomenon.  I recall hearing of things like that.  My grandmother used to say that people like that were to be pitied, because they were so unsure of themselves and felt so bad about themselves and their abilities that they must attack others they perceive as being better in some way.

Alas, Grandma Smith did not live to see the era of 24 hour news channels and everything that came of it - including the nonsense of people calling insults about those they disagree with.  Then, there was the tabloid/yellow-journalism where anyone who has a different view is berated and insulted on air.  Where personal affront replaces debate and discussion - appalling

Now, instead of people being insulting, they've turned to threatening physical violence and death.  Its appalling.

As I was reading another one of these sad accounts when I was struck by the similarity to another account. 

There was a coal-miner from Scotland in the late 1800's.  He had a habit of singing while the lift (elevator) lowered them down into the mine.  So much so that his mates suggested he try his hand at the music hall.  So he did - and the results went well.  People rather liked it.  One supporter suggested he try it at a larger venue.  He did - and was paid the princely sum of 5 shillings.  This was the first time Harry Lauder was paid to sing.

Harry Lauder?  Born in Edinbugh in 1870, he went to work early to support his mother and siblings after his father died.  He sang funny songs from Scotland and Ireland, and began writing his own songs.  Songs that people today at Scottish gatherings sing and write them off as "traditional."  Songs like "I Love a Lassie" and "A Wee Deoch an' Doris" and "Roamin' in the Gloamin'" and others that were sentimental stuff - remembering homes and ways of life that were fading even then.

He gave a "Command Performance" in 1909 for King Edward VII and 1912 for King George V.  He also toured America in 1907.  When World War I (the Great War) broke out, he gave concerts and engaged in recruiting tours - and later began singing to entertain the troops.  He sang to keep spirits up and engaged in various efforts to keep them well.

His efforts came to poignant focus when his only child, his son was killed at the front.  Captain John Lauder, 8th Argyll & Sutherland Highlanders, was killed on December 28, 1916.

Rumors began to circulate that his own men had killed him.  The investigation concluded that he had been shot by a German sniper while he was investigating unexploded ordinance in no-mans-land.  Somehow though, "witnesses" were quoted in tabloids and rumor-based papers that he had been hated by his troops.

Instead, it is likely that the people spreading rumors of the sort were acting out against Harry Lauder - the coal miner turned singer who performed for the King and was the toast of London - and these same people were striking out at the time of his greatest grief.

Harry responded to the news of the death of his son by locking himself in his room and writing.  The result was his song "Keep Right On to the End of the Road."  The day was January 1, 1917.

He kept on performing for troops and talking with them in the street and in hospital.  He did his best to carry on and do what he did for others, through his grief.

He started a fund and gave a series of concerts to help servicemen returning from war, to heal in body and spirit and return to civilian life. He raised £1,000,000 through his work and was knighted for his efforts by King George V. 

Still the rumors persisted.  He worked through them.  He carried on -

to the end of the Road.

Ev'ry road thro' life is a long, long road,
Fill'd with joys and sorrows too,
As you journey on how your heart will yearn
For the things most dear to you.
With wealth and love 'tis so,
But onward we must go.

Chorus:

Keep right on to the end of the road,
Keep right on to the end,
Tho' the way be long, let your heart be strong,
Keep right on round the bend.
Tho' you're tired and weary still journey on,
Till you come to your happy abode,
Where all the love you've been dreaming of
Will be there at the end of the road. 
 
With a big stout heart to a long steep hill,
We may get there with a smile,
With a good kind thought and an end in view,
We may cut short many a mile.
So let courage ev'ry day
Be your guiding star alway.

Chorus:

On Exploring - Where and How I Began

I was recently asked if I thought Exploratory Testing was "oversold."  I thought about it for a bit.  I suspect it has been oversold by unscrupulous charlatans as has every other approach to testing or tools for those approaches.

I've read stuff from people on Exploratory Testing (E.T.) that made me wonder if I was as absolutely clueless as I could not recognize what they were calling E.T.  Then I thought for a moment and reread some of their arguments.  It struck me that the problem was that I had no way of reacting as I kind of wanted to.  Perhaps it was because of how I came to E.T.

Many years ago I was working at a company where development managers, directors, etc., insisted that well formed and well defined tests had detailed steps to follow, with detailed "expected results" and detailed criteria for "Pass" or "Fail."

These of course were tied to the detailed requirements that were defined for the project.  These were the same requirements the design was supposed to be based on. 

Since there were not nearly enough testers to actually test the changes, the decision was made to have developers who did not work on those portions of the system execute the test scripts.  They would be familiar with the systems and be able to execute scripts quickly and efficiently.

Sure enough - they pounded through the scripts like lightning.  One minute none of them had been run, in a few hours, there were two or three left.  That was kind of a surprise since I expected executing the scripts, even with 3 or 4 people working on them, a full day and more.

Somehow, they blew through them in slightly more than three hours.

The Beginning...

Then it dawned on me - instead of looking for specific items from the "expected results" they were - I'm not sure.  So, I changed the rules.  I removed the "expected results" column from the test steps and with the next build, sent them off to be tested.

This time, it took... longer.  The people doing the testing were a bit flabbergasted - I had "changed the rules!"  And, this was not what was intended - and this "was not the right thing!"  Why, HOW were people supposed to know if it was right or not?

That was an interesting response.  It also highlighted something important - many folks are comfortable with being told what to do and what to think and when to think it.  I am not.

The issue was, as long as someone (like, me) was willing to review the "observed results" against what was supposed to be done, most people were pretty comfortable with it.  I noticed some things here though that made me wonder if there was something more I could do to introduce some variables into the process we were exercising.

I thought about mentioning it at the next project meeting.  Having coffee with a development manager, who had similar views as I did, we agreed that the "control" issues certain people had over what "good testing" looked like would never reconcile such a radical concept.

So the next round of testing, instead of detailed instructions, the developers doing testing work were given general instructions.  "Create a transaction with these characteristics..." or "Create a new warehouse item in one of these categories..."

They were instructed to write down precisely what it was they did, the sequence they did it in and what results they observed.  This worked really well.  There were detailed notes around each person's activity.  There were many areas they noted that "should be looked into."  And, most importantly in my mind, they were looking into how the system reacted.

I caught no end of grief when bosses realized I was not adhering to the standard practices. (See the blog post On Folly, I mention the company in there...)

Instead, many bugs were found that would not have been found.  We had precise "steps to reproduce" as well as snapshots from the logs and the database(s) at the time the tests were being run.  We had loads of data showing things that worked well and things that were horribly flawed.

And still, I was catching grief over this "sloppy" work that was unprofessional.  So, one day as I was feeling a bit testy, I began searching the web.  I found a reference to a form of testing that a couple of guys were advocating.

This involved not having detailed, planned scripts.  It also involved careful observation on the part of the testers and how they tested and what they did and looking at how the software as a whole behaves.

This made sense to me.  A great deal of sense.

I began reading as much as I could.  I found that this wild and crazy idea that I had, where people were not told precise steps to follow or specific formulas to determine if something is "right" each and every time they test a piece of software, had a name.

These people whose ideas I was reading called it "Exploring."  They called their approach "Exploratory Testing."   I read what I could - first of James Bach and Cem Kaner, then Michael Bolton and others.

I put "Managing the Testing Process" on the shelf, and looked at "Testing Computer Software" and "Lessons Learned in Software Testing."


Where I Stand Now...

I found one that was focused on controlling the process model and artifacts around testing, and the others to looked at testing. The two books looked at the actual, reality of testing.  That rang of truth to me.

From that point, I found less and less value in the rituals of testing.  The idea that the only thing that can be tested is "code" is one of the silliest ideas I can recall.  Yet that seems to be what some people want testers to "test." 

That is the saddest thought I can recall for some time around testing.

Has ET been "oversold"?  By some unscrupulous charlatans, yes.  By people who work with it on a daily basis?  No - Not at all.

On Folly

In her book, "The March of Folly - From Troy to Vietnam" Barbara Tuchman talks about "the pursuits by governments contrary to their self interests."  Its an exceptional book.  I strongly suggest you read it.  Here's a link.

Any software tester who has an interest in how things can be different, or at least has an inkling that things either in their organization or in general, are not working or are simply messed up really could use with reading the first two pages.  (My paperback copy is a 1985 Ballantine Books Edition of the 1984 copyright renewal)   Its good.  You could substitute "test managers" (or "software managers" or "software development leaders") for "governments" and not change that first page one iota.

Tuchman does not throw everyone who makes a mistake, or even a serious blunder, under the bus of "folly."  No, she is more precise than that.  To be painted with her brush several specific conditions must be met.

In her words "the policy adopted must meet three criteria" (sounds better that way, doesn't it?)

First, "it must have been perceived as counter productive in its own time, not in hindsight."  That is important.  After the wheels fall off, its pretty easy to track back to the "Oh, THIS should have been done differently."

Second, "a feasible alternative course of action must have been available."  OK, fair game that.  If you know something is likely to be a bad idea and there aren't any other options to try, then it's not "folly."  It often is categorized as "bad luck."

Third, "the policy in question must be that of a group, not an individual ruler, and should persist beyond any one political lifetime."  That one is a little more challenging.  How long is a "political lifetime?"  That rather varies, doesn't it?  It could be an "administration" in the US, it could be a Board of Directors configuration for a company.  It could be a "management team" structure.  It could be several things - all of which add up to a passage of time, not a quick "policy du jour."

And software - Antediluvian era

Some 30 years ago, almost exactly 30, where I was working as a programmer. For clarification, it was common practice to have programmers, programmer analysts and folks with similar titles do things like, gather requirements, confirm requirements with users/customers, create the design, plan any file structure changes that might be needed, write the code and test.  Sometimes we worked in pairs.  Sometimes there would be three of us working the same project.

The company adopted a new process model for making software.

It relied on defining requirements in advance, and hammering down every possibility and variance up front, then getting everyone to "sign off" on a form saying "Yes, this is precisely what we want/need."  Then we would take those requirements and use them for building the design of the software.  Once we had the signatures, there was no need to "bother" the "users" again.  If there were questions we'd refer to those same requirements.

Then we would use the documented requirements to drive how we documented pretty much everything else.  Our design referenced the requirements.  If we had a new screen, report or other form of interface, we could make mock-ups to show exactly what each item would look like - and how it related to which requirements.

We even had comments in the code to reflect the section of the requirements this piece of code was addressing.  We used them when building test strategies and test plans and detailed test cases.

We could precisely identify each section of each requirement then show how every section of every requirement could be referenced in every piece of the design, the code, the file structure and then in the test plan - specifically each test case.

I saw this and thought - "Wow.  This will fix so many problems we have."  The very senior person on the team, his title was actually "Senior Programmer Analyst" - he was about as high as you could go without turning into a manager, had doubts.  He was not sure that everything would work as neatly as we were told to expect it would.  I shrugged and wrote his reservations off as being "old school."

And then I tried it.

Hmmm.  Things were more complicated than anyone thought.  We kept finding conditions that we did not anticipate during the weeks of developing the requirements and doing design.  We kept finding holes in our logic.

The "good" news was that since everyone had signed off and said "This is it!" I only got into a little trouble.  The project was delayed while we reworked the requirements and got the change agreements signed then changed the code and... right.  We found more stuff that needed to change.

The folks running the initiative gently patted my hand and said "As you get more experience with the process, it will be easier.  The first few projects will have problems as you learn about the process.  Once you follow it precisely, these problems will go away.  You'll see."

That seemed comfortable.  I took solace in that and tried again.

Three major projects and - somehow - the same thing happened.  Not just for me, but all the programmers, programmer analysts, senior programmer analysts - everyone who wrote code and did this stuff ran into the same problems.

Somehow, none of us were following the process "correctly."  If we were, these problems would not be happening.

Several years later...  Deja Vu

At another company, I was now the senior developer.  My title was "Information Analyst."  I was working with some very talented people working on cross-platform technologies.  At the time, it was very bleeding edge.  Unix-based platforms doing some stuff, the trusty IBM mainframe filling the role of uber-data-server/host and them some Windows based stuff all talking and working together.  Along with code stuff, I was also mentoring/helping with testing stuff.  There wasn't a 'test team' at this shop, we worked together and some folks coded, some folks tested.  The next project, those roles swapped.  I was fortunate to have a talented, open minded group to work with.

There was a change in leadership.  We needed structure.  We needed repeatability.  We needed to fix serious problems in how we made software.

They rolled out a new model - a new software development process.  Everyone had defined roles.

We needed to focus on getting the requirements fully identified before we did anything else.  We needed to get every possible combination of conditions identified before any design work was done.  We needed to do work around codifying how requirements looked.

Then we could design software that conformed perfectly to the requirements.  We could examine all the aspects of a given requirement and handle that in out design and then our code.  If it had a new screen, report or other form of interface, we could make mock-ups to show exactly what each item would look like - and how it related to which requirements.  

Then the code could be developed according to the design, and could reference the design points and the related requirements for how the code was intended to function and what purpose it was supposed to fill.

Testing - we could build the test strategy and plan simply be reading the requirements document.  Since that was so complete, there was no reason for clarifying question to the BA's or the users or... anyone else.  Testers could sit in their cubes and design tests and then execute them when the code was ready.  Except we did not really have testers, we had developers who also did testing for projects they did not write the code for.  Except that we sometimes had a problem.

We could map out the expected results in the testing and then ask the people doing the test scripts to check "Y" or "N" if the expected results came up.

Somehow, this company, too ran into similar problems as the other company.  We kept finding conditions we had not accounted for in the detail requirements gathering.  We kept finding conditions no one had anticipated.  We kept finding holes.

When we asked about it, we were told the problem was we were not following the process correctly.  If we had, we would not have these problems.

Hmmmm.... this sounds really familiar.

The "good news" for my team was that we generally avoided being in too much trouble because everyone who needed to sign off on the requirements had done so.  There was some grumbling about we should have done a better job of identifying the requirements, but since everyone had said "Yes, these are all of them" we were able to avoid taking the fall for everyone else.

Still, it was extremely uncomfortable.

A couple years later...  Deja Vu Again

Now I was the QA Lead.  That was my title.  I was working with a small team making testing happen on code that some really talented developers were making happen.  We talked about the system, they made code and we tested it.

The customers liked it - really - the folks who used the software who did not work for the company.  They noticed the improvement and they liked it - a lot.  The Customer Service folks like it a lot.  They got a lot fewer calls from angry customers and got more of the "Here's what I'm trying to do and I'm not sure how to do it" sort of calls.  They tend to prefer those - at least at that company.

Things were working well, so well in fact that the test team was moved from that one project group of to doing testing for the entire IS Development area.  That was fine, except there were two of us for some 100 developers. Ouch.

The "approved model" for making software looked a LOT like the last one.  This time, there was the call of "repeatable process" included.  We can make everything repeatable and remove errors (I believe "drive them out" was the phrase) by being extremely consistent.

This was applied not only to the information gathering in requirements, but also in design and code development.  As one might expect, it was handed on to testing as well.  Everything needed to be repeatable.  Not only in the design process but absolutely in the execution.

So, while we strove to make these design efforts repeatable, the demand was that all tests would be absolutely repeatable.  That struck me as "I'm not sure this really makes sense" but I was game to try.  After all, a consultant was in explaining how this worked and we were essentially hushed if we had questions or doubts.

The response was something like "The first few times you try it, you will likely have problems.  Once you get used to the process and really apply it correctly, the problems will go away and things will run smoothly."

We still had problems.  We still struggled.  Somehow, even the "golden children" - the ones who were held up as examples to the rest of the staff - had trouble making this stuff work.

A few years later...  Deja Vu All Over Again

I was working at a small company.  In the time I had been there we had shifted from a fairly dogmatic approach to testing, where precise steps were followed for each and every test, to a more open form.  Simply put, we were avoiding the problem of executing the same tests over and over again, eliminating bugs in that path and ignoring any bugs slightly off that path.

The product was getting better. We had built rules to document the steps we actually took, not the ones we planned to take.  When we found a bug, we had the steps that led to it already recorded and so we could plug them straight into the bug tracker.  The developers found this more helpful than a general description.

We had documents that were needed - requirements, etc.  They were sometimes more vague than they should have been.  So we tested those as well.  This allowed us to have meaningful conversations with people as we worked to define what precisely they expected.  Of course, we carried these conversations on as we were working through designing the software and considering how to test it.

Sure, we had to sometimes "redo" what we did - but generally, things worked pretty well.  They were getting better with each project.

Then we were bought.

After the inevitable happy-sizing, the "staff re-alignment"that left us with a fragment of the old company staff, we received the instruction in the "new way" of creating software.

You start by completely defining requirements - in advance.  Nothing happens until everyone agrees that all the requirements are fully documented and are complete.  Then design happens and everyone relates the design to the requirements.  And coding is done strictly against the design to make sure everything is according to the requirements.  Testing planning is done with a specific strategy created to reflect the requirements.  Then test plans are created, from the strategy, and refer to the requirements.  The test cases are detailed, repeatable sets of instruction to make sure the tests confirm to the requirements and can be executed many times without variation.

"Yes," we were assured, "this will take some getting used to, but once you understand the new process and follow it, you won't have any problems and the software will be great."  As projects had problems, of course it was because we were not "following the process correctly."

Looking back...

The first time I encountered a process like that, I was all over it.  In my very junior programmer mind it seemed to make perfect sense.  The next time, I was wary.  I had seen it fail before - and let's face it.  If problems are the result of people "not following the process" correctly - or "not understanding" the process.  I think there may be a problem with the process.  After all, these were smart people who had gone through the training for the new way of doing things.

The third time, right.  Good luck with that.  I expressed my concerns and reasons for being unconvinced.  The last time - I rebelled.  Openly and loudly.  I broke it down with them, made reference to the model they had drawn on and showed documentation for multiple places that demonstrated that model was innately flawed.  No amount of "tweaking" would fix the central issues.

I was told, "No, this was developed for us, specifically."  I challenged that by pointing out the reference materials readily available on the internet showing this process model - complete with the same step names, artifact names and descriptions of the process.  I then explained why this model would not and could not work in their context.  (By the way, it was the same reason it was doomed in each of the previous instances I encountered it...)

Those issues are this -

* Human language is an imprecise form of communication.  People can and will misunderstand intent.
* Requirements are rarely understood, even by the people who "know" the requirements best - the people asking for the change.  People have a hard time considering all the possible paths and flows that result from a given decision.  Once they see the result, they will better understand their own needs.
* Humans do not think in a linear manner.  That is the single biggest problem I see in the "repeatable" models put forward.  At some point there is a cloud with the word "Think" present.  At that point, the linear model fails.

With each new standard model put forward, there are people working in the industry governed by the standard with practical experience around the work the standard is intended to direct and mold.

When they raise objections, dismissing them as "self-serving" is, in itself, self-serving.

Your pet project may well be ugly and unwieldy.  Admit that possibility to yourself at least, or join the list of "leaders" who commit folly - and destroy the thing they are trying to save or build.

More Than One Way to Confer - CAST2014

In August I was in New York City for CAST 2014.  This was the Ninth installment of the Conference of the Association for Software Testing.  Like many non-profit conferences, there is a mix of staff and volunteers making sure things run smoothly.

As luck would have it, a couple of weeks before the conference began, I got a phone call asking if I was available to "help out" a little more than usual.  It seems the nice lady who normally runs the registration desk was not available this year - things going on with the family and medical issues and... life getting in the way.  I said "Of course I can help out.  Not a problem!"

The result was, even though I was at the conference, I was tracking the activity by watching twitter because I was awfully busy not being in the room.  It was really kind of fun.

Now, CAST is interesting in that for the last several years we have hosted a live webstream and recorded the keynotes and several of the track sessions - and then loads them to YouTube as soon as they are able to load them.  In fact, you can see them here: https://www.youtube.com/user/TheAstVideos

So, I really wasn't worried about missing content.  While I was kind of bumming about missing the energy in the room(s) I knew I would get the highlights from friends and colleagues later that evening.  Why did that matter?

Well, CAST is interesting.  Part of the "energy" is in the portion of each presentation (including keynotes) referred to as "open season" - That is a moderated Q & A session where, essentially, questions on the presentation, the experience the theory behind it or lessons learned are all fair game.  Discussion is aimed not to dance around issues but to get to the heart of questions that people in the room have and wish to know more about.  When the time is up, it is not unusual for the discussion to move to the hallway or to an area intended for just that sort of interaction.  For me, this is one of the main attractions to CAST:  the discussion.

Another main feature is related - the chance meetings with people in the hallway or at breakfast.  Frankly, one of the best aspects for me are these meetings.  The "I just bumped into {famous tester/tester I respect}" events serve as highlights of my day and week.  This year, I admit, it was a flurry of these meetings - Fiona Charles, Erik Davis (and his crew from Hyland Software - these folks get it), Huib Schoots, James Bach, Griffon Jones, Matt Heusser, Karen Johnson, Michael Bolton, Selena Delesie, Michael Larsen.  The list kind of goes on and on.  People I knew from other places and years past and kind of looked forward to meeting again.

Then there were others I had not met in person, but whose writings I respect, like James Christie, Richard Bradshaw (whom I had met before, but never really had a chance to talk with), Smita Mishra, John Stevenson, and ... and... and ... Right - you get the idea. 

There was an impromptu gathering at my hotel bar when I happened to run into people I did not expect to see there - and then more appeared - and more - and - There were some 15 people at one point on a Sunday night, with not planning what so ever, just having drinks and great conversation and ... Frankly, I don't see that very often at other conferences.  

One thing I must admit though, working at the Registration Desk, while it is a lot of work, is also a LOT of fun.  In what other way do you get to greet EVERY SINGLE PERSON who walks in the door? 

I can hear it now - "But, I'm not good at that - I am shy and kind of introverted."  HAH!  Ya know WHAT?  Very few people are comfortable doing that - walking in to a room full of strangers and greeting people and being warm and friendly - and saying hello and ...

OK - here's a secret:  I kind of suck at that.  I worked really, really hard to do that and not come off like a jerk.  Ya know how I overcame that?  I thought to myself, "Self, how would you want someone to help you feel comfortable in a strange setting, like a conference when you may not know anyone and all these 'legends' are walking around?" 

So, yes.  This is far from the full list of people I had the chance to meet and hang with at CAST.  It is simply some of the thoughts running through my head as I think back on that week in New York. 

What is my point on this?  I think it is pretty simple.  If you have the opportunity to help out a local meet up or gathering or even a conference - do it.  If they ask if you could work the sign in/registration table - DO IT.  It is kind of cool!  You will do something others won't - say hello to every person who comes to the event.

You never know who might walk through the door.  One of them might be Opportunity.

August, 1914 and Confirmation Bias

People following me on Twitter know that I regularly, though not always, tweet something about an event that occurred in history that day.  People paying attention have noticed that this month, August, I have paid particular attention to August of 1914.  This month marks the 100th anniversary of the outbreak of World War I.  Many Americans look at this as an interesting but relatively minor footnote that only touched the US much later.  This is unfortunate.

This war tumbled empires, shattered people's concepts of surety and security, and marked the shift of the order of the world.  Former colonies soared to importance.  Australia, New Zealand, Canada and a little country called the United States all found themselves thrust into the limelight where European powers thought they alone held sway.

Much of this was due to the events one hundred years ago this month.  There are lessons we  can learn today from these events as testers and as citizens of the world.

Belgium

Much has been said by popular historians on the assassination of Arch-Duke Franz Ferdinand and his wife.  On a simple timeline, this prompted demands and ultimatums and threats - and as national figures refused to back down from the brinksmanship they played a part in creating, nations declared war on each other on a scale that had not been seen since Napoleon's near conquest of Europe.

Then there was Belgium.  The young King Albert, trained in statecraft as well as military matters feared that if war came, it would roll through his country as so many other wars had in the past, from Julius Caesar to Napoleon.  Countries that were pledge to defend Belgium's neutrality were pushing themselves and each other toward war.  Germany, France, Britain, Austria all had pledged to preserve and protect Belgium's neutrality under the Treaty of London of 1839.

When Germany ordered mobilization on August 1, Belgium ordered its forces to mobilize, with the order taking effect at midnight.  Soldiers reported to barracks, reserves were called up and vigilance was increased along all of Belgium's borders.  The standing policy and agreement was that if any country should invade Belgium, the guarantors of her neutrality would come to her aid.

Germany issued an ultimatum to King Alfred that her soldiers allow German forces to pass through Belgium to invade France.  Alfred refused.  His fear was simple, if Germany won the war, how likely were they to honor their promise of withdrawal after they violated their promise to not invade?

Germany invaded Luxembourg on August 2.  Germany declared war on France on August 3.  That same day, August 3, Belgium refused Germany's demand to allow German troops to pass through Belgium. 

August 4, Germany invaded Belgium and Britain declared war on Germany for doing so.

Albert disagreed with many of his generals who insisted that offensive operations were  key to victory over Germany.  Many modeled their thinking on the French plans, which called for massive assaults against German positions to drive Germany out of Alsace-Lorraine (lost during the Franco-Prussian War in 1870) and defeating German offensive operations by attacking German bases.

Albert insisted on defending the forts on the frontier and defending key cities as long as possible, and keeping the field army as an Army in Being on Belgian soil.  Thus, the German offensive would hit abd be delayed by fortifications, while his main forces finish equipping and preparing for battle.

Neither the Germans nor the French expected the Belgians to put up any kind of meaningful resistance.  The Belgians were, in the eyes of the "Great Powers" an inconsequential force.  They were the Hobbits of the Middle Earth of Europe in 1914.

France

French pride was injured in the short, painful Franco-Prussian War where the main French forces were surrounded and defeated in a massive double encirclement at Sedan.  It was a humiliating failure.  Since then, the French military establishment had looked forward to restoring their honor and the glory of France.

At the front of their minds was restoring to the French nation the provinces taken by Prussia after the French defeat.  They longed for the day they would march in triumph and retake their lost territory.  They longed for the day they could invade Germany and slice off a portion of German territory in retribution.

In doing so, their plans were all of the offensive.  Plan XVII called for a massive invasion of Alsace-Lorraine and then sending overwhelming numbers into Germany proper.  They would pull divisions from their territories in Algeria to make this happen, along with mobilizing as many reservists as possible.  Those who spoke of concerns about the defense of France and Paris in particular were viewed as defeatists if not out and out traitors.

Commanders spoke of élan and cran and the pantalon rouge as the keys to French victory.  If the Germans massed their main offensive to try and attach the French flank, then there would be fewer Germans to resist the French onslaught aimed at Metz.

The French Commander, Joffre, was so adamant in this that warnings and messages from Belgium on the size of the German attack into Belgium were dismissed as coming from unreliable sources.  People fighting the enemy in front of them were considered less reliable than people making plans to fight an enemy they were not yet ready to face.

Finally, a cavalry unit was sent forward to look for evidence of this massive invasion into Belgium that Albert and the Belgian commanders were frantically sending messages about.  The cavalry found little to support the claims.  This was mostly the result of effective screening by German cavalry to offset the efforts of the French.  IN short, the French cavalry failed to recognize that they were being themselves screened by German cavalry.  If there was no "massive invasion" happening, there would have been fewer Uhlan regiments present.

They saw, but did not observe.

The day after the French emissary told the Belgian high command that they were mistaken in the size and scope of the German incursion, Belgian cavalry units, fighting dismounted, defeated a large force of German Uhlans at Haelen.  Four days later, the last of the forts around Liege fell to the Germans who brought up massive artillery to destroy the defenses.

August 21, before the French attack at Charleroi could begin, the Germans launched their own massive attack.  Lanrezac succeeded in saving his army of 15 divisions, by withdrawing instead of following orders and attacking the 18 German divisions as he was facing.

Britain

In spite of joint operation plans with France, Britain was not as willing to jump into the fray as any of the major belligerents.  Instead, the Royal Navy was mobilized to protect the English Channel and keep sea lanes open.  The stated intent was to ensure that none of the navies of nations who went to war would be in a position to harm her shipping or harbors.  In reality, the intent was to help protect the French coastline and ports in case there was a need to send troops to Europe.

Where the governments of Germany, France, Russia and Austria-Hungary were  united in their desire to go to war, Britain was not.  The government face a very real threat of loss of confidence if war was entered into without an overwhelming reason.

The only way to ensure support for war was if Belgium was invaded.  The Germans did that on August 4.  Britain declared war and mobilized her army.  The German Chancellor was astounded that Britain would go to war over "a scrap of paper."  She did.

Of the countries in Europe, only Britain did not have conscription for military service in 1914.  Any force sent to Europe would be volunteers.  The British Expeditionary Force sent to France on August 7 consisted of some 80,000 men.

They met the German army in force at Mons, on August 23, three days after German forces occupied Brussels, the Belgian capital.  The British forces were heavily outnumbered, with German forces having twice as many pieces of artillery.  In spite of this, the British held the German advance and inflicted extremely heavy casualties on their opponents.

The Germans did not expect the British to put up much of a fight.  There is a tendency among armies and nations to judge opponents by how they behaved in their last conflict.  The idea that someone learned something seems a revolutionary concept.  Encountering a skilled, well trained and motivated opponent when one expects an inept one tends to shatter more than the idea that something will be easy.

It can also shake the confidence in your own abilities, despite whatever exhortations leaders make to the contrary.

By the end of August 24, the German infantry soldiers knew they were in for a harder time than they had been led to believe.  Within a matter of weeks, the people of Germany, France and Russia would know that the quick war they all expected was an illusion.

Lessons

King Albert of Belgium steadfastly held on to the idea that a free and independent Belgium needed an army in the field, holding a tiny portion of Belgium.  Without that, they would be at the mercy of the German invaders, or possibly worse, their Allies.  He also knew that if his small army could delay the German advance, he could gather support from around the world.  If he could hold on long enough, that support would manifest itself in untold millions of soldiers.  He expected his country to be brought to the brink of utter destruction by resisting.  He had proclamations issued to all the towns and villages saying to turn in all weapons before the Germans arrive, lest the owner be killed.  He expected war to be made on the civilian population.  I am not certain if he expected war to be made in the way that it was. 

Kaiser Wilhem II, Moltke and Falkenheyn of Germany all expected Belgium to not resist.  At the most, they expected a token form of resistance and described this as soldiers at the frontier firing rifles into the air and others lining the roads as the German columns passed by.  When the Belgians fired their weapons, it was anything but in the air.  They did not behave as expected.  The brutal reaction of German forces (there is no other word to describe it) in Belgium was perhaps worse than Albert feared.  He described the potential reaction to Belgian Resistance as "crushing."  It certainly was.  Additionally, they expected the British forces to fold up easily and leave the French to their fate.  They found it hard to believe that England, a fellow "Germanic" country, could really make war on Germany.

Grand Quartier Général, the French High Command, the whole thing, refused to argue against Joffre's insistence on attack.  The organizational culture refused to consider the possibility of error.  This nearly lead to a complete disaster.  They were saved by a handful of officers in the field who saved their commands and France, even as they sacrificed their careers and in some cases, lives.

General Sir John French and H. H. Asquith of Britain managed to get a functioning and capable force into action in Europe in time to prevent a total collapse of the Allied lines.  Asquith demonstrated the stereotypical British habit of muddling through toward an end.  French had a nasty habit of not letting facts confuse him once his mind was made up.  However, both managed to get a coherent force into place and gain enough time for a larger, more substantial force to become available.

This was partly through the efforts of Lord Kitchener, the first serving officer in the Cabinet since the time of Charles II, he became Secretary of State for War (and the image of recruiting posters with his stern face looking out with the motto "Your Country Needs YOU."  He also horrified members of the War Council the first day by saying that Britain needed 70 Divisions, not the 6 that were available in August of 1914, and the current professional force should train the new recruits.  He also said it would take at least 3 years to get that number of adequately trained solders ready.

King George V of Britain played his own part in keeping focus on what was needed.  By calling for protection of "small nations" against invading hordes (actually, "Huns" which was the term Wilhem II used in reference to his own army - he could have chosen a better word, but landed there time and again) King George demonstrated his own model of courage, even when it was his children and relatives who went into harms way.  (Much can be taught to today's leaders in many countries, I think.)

Lieutenant Maurice Dease, V.C., 4th Battalion, Royal Fusiliers who manned a machine gun at Mons when all solders in his section were killed or so severely wounded that they could not handle the weapon.  Only after being wounded five times, when he was unable to operate the weapon, would he allow himself to be evacuated to a hospital.  He died of his wounds, but helped save his Battalion and gained the thing the British needed most right then: time.  

Private Sidney Godley, V.C., 4th Battalion, Royal Fusiliers, who took over from Lt Dease after he was mortally wounded, and continued operating the machine gun for 2 hours while the Fusiliers, and the rest of the BEF retreated.  He continued to do so, despite being twice wounded, until he ran out of ammunition.  He then dismantled the gun and threw the pieces away, to prevent them being captured.  He spent the rest of the war in a German POW camp.

And Testing...

Each of us has our own biases and beliefs. We have the choice of working hard to set aside those biases and examine the evidence in front of us, or we can dismiss the evidence as wrong or from unreliable or irrelevant sources.

We have the choice of looking at what is , what an impartial observer might note, or what we wish it to be.  In the end, testers are bound to observe how software is currently functioning.  Then we can ask two important and related questions:

Is this what I could reasonably expect to see?

Is this a problem?

After we consider those, we can then provide a reasonable evaluation of the software.  If people do not want us to consider these questions, they are taking on the role of the various command structures from August of 1914, who could not bring themselves to believe what was unfolding in front of them even as their plans and dreams of glory ended in the bloody mess at the Marne.