Thursday, October 16, 2014


Thursday morning at StarWest is lovely.  Some clouds, comfortable temperature (for me) and we started out sitting drinking coffee and talking about testing.  What a great start to the day.

Heading off for some day-job work stuff, will be back for the morning keynote.


So, the work stuff took longer than I thought it would, so I'm a bit late to this keynote (which is disappointing to me as I wanted to hear it.)  Alas - 

Ben Simo is a really nice guy, mild and soft spoken.  Do not be mistaken into thinking he is anything but an extremely skilled  describing his experience trying to get health insurance coverage through as "The Power of an Individual Tester."

Summary - It ain't pretty.

You can see his thoughts, comments and learnings here: This is different from his regular/normal blog, here:

The problems were Legion: poor security questions, poor account verification handling (as in "We sent you an email with your new password!" except it can take 3 days to get delivered and the link for the password expires in a couple of hours.

Then there were problems with how information was returned and displayed - anything from sensitive data and it personal information posted behind the screens, not visible - unless you trace what is actually being sent to the browser.

The "skilled hacker" comment came as a result of Ben noticing stuff and looking to see how far the thread went.  With a series of fairly simple steps, a person with bad intent could get everything they really needed simply by monitoring cookie streams, etc.

The data included in what was being sent to the front end included income and other sensitive data that should not really be returned to the browser in a secure system.  There were other issues around when and how data was collected from the screen, which is retained and passed back and forth.

Lesson: Technical people - be straight when talking with non-technical bosses.  Be absolutely certain that their understanding of how the system works is correct.

When there is bad news - be very precise.  Don't get involved in turf wars, give people information and be straight.  No matter how BAD the information is, get it out and then figure out how to deal with it.

Ben gives an excellent overview of problems in the site - starting with fundamental design flaws through transaction handling that had clearly not been considered.  The probability of success was low for such a complex system - then they made it worse by not making certain that public officials could give real, accurate information about the state of software.  Whether they chose to use it is beyond your control.  Be honest.

Ben does note that there are now some improvements - for example the actual ability to send feedback - and the form actually opens.


Next up – Rainmaking for Testers (and managers) with Julie Gardner.  She gave the opening keynote yesterday.  She’s with RedMind in the UK.

Julie talks about “trusted adviser” and “rainmakers” – and getting the image of what a “trusted adviser” has to say.  The issue is – what makes an adviser trusted?  How do people view you?  Are you the guy from Office Space?  How about a used car salesman?  How about Wormtongue from Lord of the Rings? 

Each of these were trusted, to some level, by some people.  They were not trusted by others.  This depended on several things, including the way that messages are delivered. 

It is important to be straightforward.  It is also important to frame bad news in a clear way – not overstating the problems, not understating the problems (note Ben’s keynote) – but give a concise evaluation.

Be willing to help, even if it is hard an outside of your normal realm, role and behavior. 

AND! She used one of my favorite words!  “Your manager does not want you to be a sycophant.” 

Don’t be a “yes man” a “suck up” – be straight with them.

(Pete Comment: OK – slippery ground here –)
What does “development” produce – CODE!

What does “testing” produce – (varied things shouted) Information – Documentation – Bugs

In general, too much detail will produce information overload, which will distract people from the message you want to deliver.  Be succinct.

Trust entails risk.  To be trusted is not a right – it takes time to develop trust over the questions at hand.  Speaking clearly and making sure that “the proof of the pudding is in the taste” – Make sure you deliver on the words you choose to use – the things you say mean things.  Make sure that when you say something – you can actually deliver.  People are taking on a risk by trusting others. 

What gets in the way of trust?  Bad manners – dishonesty – exaggeration – arrogance – empty promises – on and on and on…    Don’t do that…

Building strong relationships require you to ask questions and then listen to the answers – carefully.
Concentrate – listen (you have 2 ears, & 1 mouth – you also have 2 eyes – listen with your eyes as well. 

When you need help – ask.  You can’t be perfect for everything.  You can be empowering someone else by recognizing them.

Give advice effectively – Be prepared (as much as you can).  Advice is rarely a perfectly logical process – so be straight, don’t hide bad news and don’t forget the good news.  Don’t bury the bad news, be balanced when you can.

Know your audience – different people need different degrees of information.  Don’t give them the control panel for a jumbo jet, when the manager needs the information on the car dashboard. 

Now she’s talking about predictions- Number of tests you have to run, the number you expect to run by X time frame.  Then there is the question of predicting bugs… (Pete comment: can you really predict bugs? How?)  Possibly in some instances the information found in testing over time may help. (Julie refers to this as “defect measurement rate” which was developed by a colleague of hers.)
(Pete Comment – I know people have different views – I am not convinced that predictions of bugs to be detected is terribly helpful.)

OK – Moving on – Developing a helpful mindset is important.  Don’t place blame, don’t slag someone for not being perfect (you aren’t either.) 

Julie had a survey – “What do senior managers look for in a test manager?”

Top 10 common responses:
10 - Pragmatic
9 - Understand Testing
8 - Identify and Manage Risk
7 - Cooperation and Team Players
6 - Understand the issues and the politics around them
5 - Trend Analysis and forecasting
4 - Flexibility
3 - Abe to communicate at all levels
2 - Honesty & consistency
1 – Understand the Business

To move from “trusted adviser” to rainmaker – consider:
People do business with people because they choose to, not because they have to.  We can always find others doing the same thing or selling the same product.  It’s the personal connection that makes that happen.”

Even when marketing yourself – retain your integrity. Don’t “sell out.”

Testers can be the “Jiminy Cricket” for the team, project, organization, and act as the conscience.  We can be honest with people and give straight information – and ask if we are doing the right thing.  We must have the courage to do that – and speak truth, even when people may not want to hear it…
Adding the elements of “trusted advisor” with credibility, you get integrity.

Annnd – power is gone – it was a very good session…
Hallway (well, coffee line) conversation with a couple of people on testing, exploratory testing, applying ET in their environment and leadership.  Wonderful!
LUNCH!  Be back later with updates from James Christie's presentation...
Ummm- one thing - we don't have wifi in most of the session rooms - and limited power sources.  So, updates come as I can post them -

We’re BACK! 

After a lovely lunch table conversation with many people, including Griffon Jones, on Exploratory Testing, I’m now in James Christie’s session on “The Unfortunate Triumph of Process over Purpose.”  

James is best known, lately, for his opposition to the new ISO 29119 standard on software testing. I rather find this unfortunate – not his opposition, but that most people see him as a controversial figure.  I can say, after many conversations, James is an excellent thinker, writer and has very good ideas.  I have never heard him present and I am looking forward to this.

His talk is based on his career as a (process) auditor test manager, with examples from specific projects he worked on.  One example, the first, was a project done for a division of the British Government.  This project was politically sensitive at the time, and included significant expectations around accessibility (visually impaired in particular.)

In a sense, this was normal.  The only problem is that when you have a risk-averse organization the likelihood of failure tends to go up.  Instead of “Keep Calm and Carry On” the motto tended to be “Keep Calm and Ignore Reality.” 

ANNNNNNNNd… James refers to non-existent documentation – 70 pages of stuff that made no sense.  James role was to document and ensure that the items worked as intended.  Thus he made lovely documents, but much were of little value. 

Test plans were irrelevant because the documented requirements and the software developed had little to do with each other.  There was not a rejection of any form of Agile methodologies, they simply did things the way they always had.  (James is speaking really, really fast – banging points out – it is hard to keep up.)

The project was declared a success, even though they had just barely delivered something that sort of worked. 

By comparison, the crash of banks in 2008 resulted in several findings – the Parliament report on the failure of Royal Bank of Scotland had a dramatic point.  (See photo) It is an excellent summary of what happens when people worry about what the steps of the process are, and forget about what happens around the reason the processes exist. 

Thorstein Veblen described what he called “trained incapacity” (1914) where people are given such detailed, proscriptive methods that when the situation changes, they cannot adapt.  People are trained to not be flexible, and the result was people who were easily and often bypassed because they have been locked into one and only one way of working.

Isabel Meznzies Lyth (1959) described problems resulting from organization’s structure and processes: Task, Technology amd social/psychological needs of the people.  Lyth was looking at nurses, and how the growing view of them being fungible resources, thus nurses were seen as interchangeable. 

The problem was that these changes were not helping the people who needed to do the work, they were designed to help managers manage those people. (They also led to entrenched, fixed thinking similar to the Maginot Line built by the French in the 1930's against a German invasion.)

David Wastell demonstrated that structured methodologies did not actually advance the job, but they served as social defenses for the managers.  They showed no value in and of themselves.

James slides in to the ideas around “Rules versus Principles,” citing FD Roosevelt, “Rules are not necessarily sacred, principles are.” Principles are what we use as fixed points to hold ourselves – we use them to measure ourselves against our own values. 

Brenda Zimmerman describes the difference between complicated and complex.  Consider the space program, compared to raising children.  We can follow the same steps and get people to the moon and back.  We can follow the same steps with each child and, somehow, come up with completely different"results."

Cynefin – Welsh word meaning “whole man” from Dave Snowden.  This is a framework to help people make decisions and chose a response.   Hence, “complicated” situations are less prone to repeated processes (best practices) because, like raising children, it is difficult to master and predicting outcomes are pretty much impossible.

Snowden’s point is that there is no real clarity to what is needed. 

The center of the diagram is disorder.  This is where we normally are.  Most people will try and find a way to migrate to a situation that is easier to understand so we tend to impose those values from “Obvious”

The boundary between Chaotic and Obvious is a mess.  You can’t really balance between them  People who find themselves on that boundary are, as Snowden describes, in an “Oblivious” state – they think they are in Obvious, but are not – and will tumble into Chaotic. 

Most of the time, software projects are bouncing between Complex and Complicated.  This leads us from believing that “best practices” will save us – until we realize we are actually dealing with Chaos.

When you deliver software in spite of the process, you are subverting the process, not adhering to it.   
It is the crash into reality that ends the delusion.

James cites Wenberg’s “Second Law of Consulting” – “No matter how it looks at first, it’s always a people problem.”

(Pete note – this is around the 20th time in the last 4 days that a different Weinberg reference was made – all different quotes – and just the ones I have heard, or made myself.  Thank you, Mr. Weinberg, for being so inspirational.)

“Any approach to testing that ignores “people problems” and tries to tame human nature with rules, standards and rigid processes is doomed”

Thus, any attempt to force people into a box will fail.  Full stop.

When you are so terrified of something, unless you focus on the people, you are almost certainly going to bring your worst nightmare true.

Questions – first one is around what does one do to respond the people who will say things that oppose all standards and processes. 

James’ response is that people don’t understand.  He also clarifies the difference between standards and conventions.  These are similar – but not the same.   (Pete comment: and this is a simple concept many people fail to realize.)

Well done, and well said, James. 
Right - after a brief break and more hallway conversations, I'm in the room for Paco Hope's keynote "Softwarts: Security Testing for Muggles" which he refers to as a course in "Defence Against the Dark Arts.Black Hats"  Paco has been walking about the conference off and on the last 2 days wearing... I think they are "sorcerer robes?"  Except given the title of the presentation, I'd expect they'd be more "Harry Potter" style than "Sorcerer's Apprentice" style.  Ah - well - its all good.

We begin with prizes from the test lab.

And Paco is off to a running start... well - maybe not running.  He starts by admitting that he is "mixing genres" which is kind of appropriate for security testing.  The thing is, it is hard to identify who the good guys and bad guys are - without some work.  They don't really wear black hoodies when they are doing their thing (to the best of our knowledge.)

Generally, security defects are not that different than other defects.  They CAN be different, but often times they land in the realm of unintended consequences.  "We meant to do X (which is good) except Y also happened (which is bad.)"

The thing is, just because the security defects appear to be different does not mean that they automatically are.

Similarly - there is a myth that "security testers" are inherently different.  They do stuff that no one else can understand, let alone do.  Its like Tom Cruise in Mission Impossible hanging in the middle of the room to get to the keyboard.  Except he's probably a "functional" tester that need to climb into this silly rig to trigger some branch of the code.

There are things functional testers deal with that security testers do not need to (like "code coverage".)  There is an idea that security guys have this mystique - this wand or something.

Except testers actually have test inputs & harnesses.  They have user stories, use cases and documented requirements (which serve as the magic map of the kingdom.)  And there are logs and profile info that can act to support you.

And he relates a story of how he found a hole in a system dealing with interest rates & large purchases (financial stuff.)  He found a hole in the system where he could change the rate (increase) in the confirmation message.  He thought this was a big deal - Apparently it was not to them (of course, they never checked to see if

Four Principles -
1 - Orcs not Elves
An Orc is a dumb brute with a primitive weapon and you can point them somewhere and something dies.  One orc is not a problem - 50,000 orcs could be.  Think - bot-nets which can be launched in denial of service attacks, or other malicious manners (and if you know where and how, you can rent time on one of these without actually needed to build it yourself.)

Offline attacks are powerful instances that allow an attacker to brute force a system - sure its hard.  BUT - it can be done.  Think, sniffers getting encrypted versions, decrypt them (brute force) and walk right in to the account.

Orcs can be applied to our own intents.  Like, write a program to create input daya; check it works, run an attack to see what is happening.  Can you withstand the demo attack you created?

2 - No Gold Required
You don't need to take your bag of gold pieces somewhere to buy magic devices.

There is stuff like OWASP with tools and resources available - free.  There is CVSS (common vulnerability scoring system) to help score security risks.  There is Kali Linux - which is pre-build and bootable.

These tools are pretty useful - people will help you - and it does not really need that much money to make happen (many are free, or can have help obtained for the cost of beer.)

You can get this stuff by doing a little bit of work.  Sometimes you can get information from various sources - like the experts themselves - via Twitter, mailing lists, things of that nature.

3 - Reverse Alchemy 
Instead of taking ordinary stuff and turn it into gold, we are going to take gold and turn it into ... crap.

HTTP proxies are powerful tools we can use - to do our thing - by watching what happens when information passes through it.  Pretty much everything runs through proxies - why not set up one of our own?

Secure connections? Well, if you're running your own proxy then you can see what is happening.  Cool, no?

This allows us to monitor, intercept then rewrite traffic in your own proxy.  You can capture stuff and tweak it.  The data you change is now what you want to exercise against, or through, your system. 

4 - Use a Spell Book
You don't need to memorize everything - have a spell book to help you recall the right command for fireball (or whatever.)

Keeping a reference handy for what you need to do - how you can make things happen by retaining links, setup information, whatever.  Spells -> Commands; Animating the Dead -> Simulations, and so forth.  The specific things you need to do are the contents of your spellbook.

Equivalence Class partitioning may relate to XSS or SQL injections - these may be "classes" of attacks or hacks.

Think about the things that "can't happen" or places where "this could not happen."  It is rare that systems have no vulnerability.  EG., a 3rd party service provide sending data can (and will) make mistakes.  That is a vulnerability.

Combine those ideas and voila!  You can begin to be a security wizard, too!


Some announcements - and I'm almost out of power.  more updates and pictures will be loaded.
And - pictures loaded - including this one of the ice cream that was available for the afternoon break on Thursday....  Yes, for those who have not been to the Disneyland Hotel, Mickey is impossible to escape or hide from (look at the pattern in the carpet.)


About to head to the airport.  I intend to post a summary/retrospective of my week from there.


  1. Thanks Pete, but I was the test manager, not a process auditor. I have never been a process auditor in my life (I'm proud to say!).