Agile Testing with Lisa Crispin

Marekj (@rubytester on Twitter) whipped up an awesome graphical explanation of the Agile Testing Quadrants, along with an animated sketch. If you’re new to the Quadrants (aka Agile Testing Matrix from Brian Marick), these will help you grok their purpose and how to use them. The slide deck contains references for further learning on the Quadrants. Many thanks to Marekj for this contribution to our agile and testing community!

I’ve been writing a lot on SearchSoftwareQuality and Techwell (aka Stickyminds) lately, no time to blog. I have lots to say so I hope I have blogging time soon! I’ll start by reporting back from Belgium Testing Days.

There is so much goodness to report from last week’s Belgium Testing Days! I’ll be writing all that up, but I wanted to start with the great ideas that came out of my session on “Speaking their language: What testers can learn to work more effectively with programmers”. My slide deck will give you a bit of a feel for the talk, but as I’m going with simpler slides these days, you had to be there to know what I said. However, much more interesting are the ideas generated by the participants, who divided into small groups and made lists of what testers should learn to help them communicate and collaborate better with programmers. I asked them to go beyond the ideas I described in my talk. Each group gave their top choices, and I wrote them on a flip chart.

The last group to go had the one we all agreed was most important – have fun!

My friend and excellent agile coach Michele Sliger, co-author of The Software Project Manager’s Bridge to Agility ,  recently emailed me this question that she hears a lot: “Where should I start looking to learn more about test automation and tooling?”

Naturally, I replied with a typically long-winded answer. She must have liked it though, because she suggested I cut and paste my reply into a blog post. Here it is!

The key thing with a newbie agile team is to not say “move testers on all teams into automated testing” but “help all teams learn to take whole-team responsibility for quality and testing, and learn how programmers and testers collaborate to automate tests at all levels”. Automated test code is code, and generally, it’s much quicker and better in the long run for programmers to code the automation fixtures. Testers know what to test, and when programmers do the coding, testers have lots of time to help customers come up with examples of desired behavior to turn into tests. Then they can pair with the programmers to automate regression testing and any other types of tests (performance, load etc). Then they will still have time to do the all-important manual exploratory testing.

If they read our book Agile Testing, they will learn that they have to be patient and invest a lot of time in learning and experimenting with different test frameworks. And they should use Mike Cohn’s test automation pyramid concept to see where they’ll get the best ROI in automation. Generally, they should always start by learning how to do TDD and get traction on unit level tests. Then they can move on to API or service level tests, then GUI. But that’s not a rule, each team has to figure out what’s best for them.

I highly recommend Gojko Adzic’s Specification by Example book, and also his blog, gojko.net. Markus Gaertner has a great ATDD book coming out but it’s not published yet.

There are SO many test frameworks, drivers, tools, it’s overwhelming. Recently we had to find a way to automate GUI regression tests for our new code that uses Dojo. Our existing GUI tools aren’t able to interpret the Dojo JS properly. It was truly a team effort, though not a smooth one. Our sys admin had earlier played around with Selenium, and we thought Webdriver might work. The sys admin did a spike and proved it did work, then he and our senior architect spiked a couple of different frameworks – one homegrown, one using Geb. They demo’ed it and got everyone’s opinions. For now we are going with the Geb framework, but down the road we might decide it’s not quite right and try something else.

Anyway, that’s a long-winded answer. One good place to start getting an idea about functional test tools is the Agile Alliance Functional Test Tools group’s spreadsheet comparing various tools, at http://bit.ly/AgileTestTools. When the teams are ready to think about functional testing, they might take a look and get an idea of all the choices. It’s a good idea to stop and plan a test automation strategy.

It is really, really hard to get started with test automation. There’s a period where it’s just extra work and no reward. But eventually they’ll cross over that Hump of Pain and start reaping the benefits, eating away at their technical debt!

Someone on the agile-testing Yahoogroup mailing list posted a link to a blog post in which he proceeded to misuse, maul and maim the Agile Testing Quadrants. There is no way to put comments on that blog post to try to refute his claim that the quadrants are somehow a waterfall process. Since other people might misunderstand the purpose of the quadrants, I’d like to put a quick explanation here.

Agile Testing Quadrants

You might want to start with Brian Marick‘s original posts on his agile testing matrix, which we called the Quadrants and (with his permission) made the heart of our Agile Testing book.

The quadrant numbering system does NOT imply any order. You don’t work through the quadrants from 1 to 4, in a waterfall style. It’s just an arbitrary numbering so that, in our book and when we are talking about the quadrants, we can say “Q1″ instead of “technology-facing tests that support the team”. 

Most projects would start with Q2 tests, because those are where you get the examples that turn into specifications and tests that drive coding, along with prototypes and the like. However, I have worked on projects where we started out with performance testing (which is in Q4) on a spike of the architecture, because that was the most important criterion for the feature. If your customers are uncertain about their requirements, you might even do a spike and start with exploratory testing (Q3).

Q3 and Q4 testing pretty much require that some code be written and deployable, but most teams iterate through the quadrants rapidly, working in small increments. Write a test for some small chunk of a feature, write the code, once the test is passing, perhaps automate more tests for it, do exploratory testing on it, do security or load testing on it, whatever, then add the next small chunk and go through the whole process again.

The quadrants are merely a taxonomy to help teams plan their testing and make sure they have all the resources they need to accomplish it. There are no hard and fast rules about what goes in what quadrant. Think through them as you do your release, theme, and iteration planning, so your whole team starts out by thinking about testing first.

Michael Huetterman adds “Outside-in, barrier-free, collaborative” to the middle of the quadrants, see his Agile Record article or his excellent book Agile ALM

Visit my Presentations page for some slide decks that contain more information on the quadrants, or check out our book. I’m always happy to talk about the quadrants, just send me an email!

I have been doing some writing in the past six weeks, in addition to rehabilitating my broken-and-now-healed ankle. But it’s all been published elsewhere. Here are some links:

Software Quality Connection, “How to Improve Communication between QA and Development“

SearchSoftwareQuality.com, “Agile development: The whole-team approach”

Also on SearchSoftwareQuality.com, “The whole-team approach to Agile development: Examples and benefits“

I’ve got a bunch of tips and Ask the Expert Q&A on SearchSoftwareQuality.com, please check them out! I’d appreciate feedback.

My review of Uncle Bob Martin’s The Clean Coder is up on my Techwell blog. If you’ve read the book, I’d love to hear your thoughts on it. If you haven’t read the book, please do so!

My interview with Simon Baker was posted on Stickyminds and Techwell today. There will also be highlights in the next issue of Better Software.

My summary of the WAT2 conference this past May, in Durango, appears on the SearchSoftwareQuality.com website (you have to register to see content, registration is free, I apologize that it’s a bit of a pain to have to register). I was inspired by this year’s theme of style and grace, as well as the amazing participants, leading testing practitioners from the U.S., Europe and Israel.

I’m honored and grateful to be the first interviewee in Yves Hanoulle’s “Who is…” series. Do any of my answers surprise you?

The Real Me, with Chester and Ernest at the Brewpub

I’m looking forward to learning about lots of people in this series!

I was pleased to learn that my StarEast talk on an agile approach to defect management generated more discussion on the topic at the conference. Please read Gojko Adzic’s new post “Bug statistics are a waste of time” – it shatters a lot of illusions that organizations treasure about defect tracking and bug metrics.