Agile Testing with Lisa Crispin

My friend and excellent agile coach Michele Sliger, co-author of The Software Project Manager’s Bridge to Agility ,  recently emailed me this question that she hears a lot: “Where should I start looking to learn more about test automation and tooling?”

Naturally, I replied with a typically long-winded answer. She must have liked it though, because she suggested I cut and paste my reply into a blog post. Here it is!

The key thing with a newbie agile team is to not say “move testers on all teams into automated testing” but “help all teams learn to take whole-team responsibility for quality and testing, and learn how programmers and testers collaborate to automate tests at all levels”. Automated test code is code, and generally, it’s much quicker and better in the long run for programmers to code the automation fixtures. Testers know what to test, and when programmers do the coding, testers have lots of time to help customers come up with examples of desired behavior to turn into tests. Then they can pair with the programmers to automate regression testing and any other types of tests (performance, load etc). Then they will still have time to do the all-important manual exploratory testing.

If they read our book Agile Testing, they will learn that they have to be patient and invest a lot of time in learning and experimenting with different test frameworks. And they should use Mike Cohn’s test automation pyramid concept to see where they’ll get the best ROI in automation. Generally, they should always start by learning how to do TDD and get traction on unit level tests. Then they can move on to API or service level tests, then GUI. But that’s not a rule, each team has to figure out what’s best for them.

I highly recommend Gojko Adzic’s Specification by Example book, and also his blog, gojko.net. Markus Gaertner has a great ATDD book coming out but it’s not published yet.

There are SO many test frameworks, drivers, tools, it’s overwhelming. Recently we had to find a way to automate GUI regression tests for our new code that uses Dojo. Our existing GUI tools aren’t able to interpret the Dojo JS properly. It was truly a team effort, though not a smooth one. Our sys admin had earlier played around with Selenium, and we thought Webdriver might work. The sys admin did a spike and proved it did work, then he and our senior architect spiked a couple of different frameworks – one homegrown, one using Geb. They demo’ed it and got everyone’s opinions. For now we are going with the Geb framework, but down the road we might decide it’s not quite right and try something else.

Anyway, that’s a long-winded answer. One good place to start getting an idea about functional test tools is the Agile Alliance Functional Test Tools group’s spreadsheet comparing various tools, at http://bit.ly/AgileTestTools. When the teams are ready to think about functional testing, they might take a look and get an idea of all the choices. It’s a good idea to stop and plan a test automation strategy.

It is really, really hard to get started with test automation. There’s a period where it’s just extra work and no reward. But eventually they’ll cross over that Hump of Pain and start reaping the benefits, eating away at their technical debt!

My teammate Tony Sweets (sys admin, programmer and genius) just posted a terrific writeup on StickyMinds/Techwell of how our continuous integration and build process has evolved over the years. This includes a history of the hardware we’ve used and how the builds have been set up, both in CruiseControl and Hudson, and all the different things the build process does.

One of my favorite excerpts:

This method worked too well. The testers started splitting up their test suites and were creating new build jobs left and right mainly because it was so simple. They didn’t have to deal with hostname issues and whatnot. They simply had to create the new job from an existing Hudson job and change whatever they needed in order to run a different set of tests. When the job ran it did not care what else was running because it was running in its own separate environment.

I love the easy-to-use Hudson UI, and this freedom to configure our own test jobs the way we want. It’s also easy to stop jobs or kick them off when we want.

Whenever I speak to a conference session or user group meeting, I always tell people, “If you aren’t doing continuous integration now, go back to your office and drop everything and get your CI going. It isn’t hard to do, there are a bunch of good tools available to help, even Testify Wizard to help you set it up. A programmer can do it in a matter of days or less. There’s no excuse to not do CI.”

I’m convinced that in 5 years at the most, any team not doing CI will be looked upon the same way we look upon teams that don’t do source code control. It would just be crazy to not do it! Automated tests don’t have much value if they aren’t giving you quick feedback several times a day. Without CI, your technical debt is bound to bury you quickly.

If I had to pick one reason our team has been so successful the past 7 years, our CI process is it. It’s the pulse of our team, and if it stops (as it did a few weeks back – see Tony’s blog post!), we all just about have a heart attack! When it’s ticking along, we feel healthy and happy.

During a flurry of tweets and blog posts about whether or not ATDD is a good thing, I posted a blog on Stickyminds with my opinion. I’ve gotten some great comments, please add yours!

I meet more and more teams, both within my company and externally, who are getting really good at agile testing. They’ve adopted a Whole Team approach, they write tests first, they collaborate and communicate well, they use feedback to improve. Agile teams are functioning much better overall than what I was seeing a couple of years ago.

One perennial issue – and this isn’t new with agile – is that testers and teams still struggle with how to design and write automated tests that have a good return on investment. That is, the tests are reliable, stable, maintainable, and provide good value for the amount of work that goes into them. We also all struggle with how much to automate, what to not automate, what to keep in our regression suite.

Janet and I devoted a section of our book to automation, and we presented principles, techniques and strategies that will help. We also include this in our Agile Testing course, and we both do conference tutorials on how to succeed with automation in agile projects. I think in addition to that, a lot of people need some practical hands-on experience. They need to be guided through an example of automating tests for a story, then they need to try it for themselves.

I’m developing an internal class for people at my company, and we also have ideas for workshops and other ways for teams who have solved some automation problems and developed good practices to share with other teams. I just mind mapped the ideas we have so far. At Adam Goucher’s suggestion, I’m posting my first draft here.

It’s my hope that I could extend this to a more generic class that would be appropriate for a conference tutorial. I’ve talked with some folks about collaborating on something like this and kind of making it open source, with several of us who could teach it, like the Watir tutorial that Bret Pettichord, Charley Baker and others have taught. But, others better than I have tried to teach hands-on courses like this and run into issues. What tools would we use that can easily be installed on all participants’ laptops? What if the participants have different skill levels? So, I don’t know if this is really possible. I’m interested in ideas, or if anyone has done this successfully.

Actually I’m sure this has been done successfully by people like Elfriede Dustin who have been specializing in test automation for years – I need to get her book out and re-read it. I’m not trying to steal anyone’s idea or say that this is a new original idea I thought up. I just see a gap in many teams’ ability to create the ‘right’ automated regression tests.

So, here is the mind map, you’ll have to open it in a new window (I’m sure there’s a better way to do this but I have to catch a plane), feedback welcomed.

Last month I had the great pleasure to participate in Agile Testing Days. I did a tutorial on using the Agile Testing Quadrants for test planning, and a keynote on 7 key success factors for agile testing success. More importantly, I got to attend a lot of great sessions by leading agile testing practitioners.

There were so many good sessions and I’ll only mention a few here. Mark your calendars for next year’s Agile testing Days, Oct. 4 – 6.

My slogan after this conference comes from Dave Evans (and from Mike Scott – not sure which one of them coined the phrase but they both talked about it): Respect the Tests. More about that in a minute.

Elisabeth Hendrickson always inspires me, and her keynote didn’t let me down. I’ve heard her definition of agile before, but it bears repeating: Delivering value in the form of releasable software at frequent, regular intervals at a sustainable pace, while adapting ot the changing needs of business. She quoted Mary Poppendieck as saying, if you want to go very fast, you have to be very disciplined. Sustainable pace means that we keep technical debt down so that the code doesn’t impede us.

Elisabeth presented her four sources of technical risk: Ambiguity, assumptions, dependencies and capacity, and her 7 key testing practices: TDD, ATDD, Exploratory Testing, Automated System Tests, Collective Test Ownership, CI, and one more that I think is automated unit tests (I can’t read my notes!)

Elisabeth also quoted Tobias Mayer, who refers to the Product Owner as the Voice of the What, and the dev team as the Tribe of the How. I like this, but I take issue with it a bit. The Poppendiecks have influenced me to experiment with having the dev team learn the business inside out, so the dev team itself can help the business prioritize and even think of features. This worked well for my team at ePlan, and I have to agree that funneling all the business requirements through the Product Owner has a lot of drawbacks.

What was new to me in this talk was something that several of us had talked about a night or two before, the idea of Collective Test Ownership. This ties in with the Whole Team Approach. Everyone on the team takes responsibility for quality, testing, and test automation.

Another great session, again from someone whose work I’m familiar with, was David Evans‘ talk. One big takeaway for me was this: If a test is worth writing, it’s worth automating, and it must always pass in the future. I often see teams that ignore failures in their CI tests because “We have to get these new stories out, this is the PO’s priority.” So the PO doesn’t care whether what she prioritized in previous sprints still works or not? When the CI breaks, the team should stop the line and get it green.

Dave pointed out that if all tests pass, you don’t need extra metrics. Defects are evidence of missing tests. So don’t call it a defect – call it a failing test, write the failing test and fix it right away. If the customer really doesn’t want it fixed now, write a story for later.

Dave recommend fixing time and quality, but don’t fix functional scope. If time is running out, thin the requirements. It’s not like leaving a wheel off or leaving out the engine. Respect the tests – any test worth writing is worth running all the time.

Dave presented his Agile Quality Framework. I won’t go into all the details, but one of my favorite parts is applying balloon patterns. An empty balloon is still a balloon, but if you don’t have the rubber outside, it’s hard to have the air and then add the rubber later. Do the rubber first – the CI, the test framework.

Mike Scott elaborated on the balloon pattern in his demo of Testify Wizard. In one hour, Mike went from zero to a full CI process, including the project two continuous builds, one running unit tests, and one running FitNesse tests. It even generated code coverage reports! I think he did this with both Java and .Net during the same session. He did this with a couple of pushes of the button,some minor configuration and a wave of his crutch.

This shows how easy it is to respect the tests. It’s not that hard to build a framework to integrate our code and run our automated tests so that we get quick feedback all the time.

Weekend before last I had the great pleasure of participating in the Austin Workshop for Test Automation. Test tool developers and users exchanged experiences and ideas. You know that automated testing has come a long way when you see the approaches organizations such as Opera and Expedia use to overcome massive testing challenges.

One discussion that helped a penny drop in my brain was the concept of test libraries vs. test frameworks. A library is a reusable component. Watir is a family of Ruby libraries that drive the browser. A framework is a set of libraries that gives you everything you need to test. Cucumber is a test framework that supports behavior-driven development using plain text. Taza is a test framework that uses the Watir libraries and generates RSpec test assertions. At least – I think so! Please correct me if I’m wrong.

A good analogy someone came up with at AWTA is that Kevin O’Connor’s (host of This Old House) tool shop is a framework – it has everything needed for any project, whereas Marek J‘s garage is just a bunch of tools. This is not to cast aspersion on Marek J’s own cool test automation framework, Watirloo.

What I learned is that my team has our own test automation framework using Watir, which consists of our Ruby scripts and test/unit. The BDD frameworks such as RSpec and Cucumber seem to be the cooler thing to use than test/unit, but I don’t know how we would take on the huge task of refactoring all our Ruby/Watir scripts to use a different test framework.

That doesn’t mean we shouldn’t look at some of these newer frameworks to see if they have something to offer that our current automation solutions don’t include. My fellow tester Lori Bergan has been looking into Cucumber. I’ve been intrigued by BDD since last year’s CITCON Denver. I like tools that seem natural.

What tools have you been trying lately? Are you happy with the ones you currently use?

Tomorrow I’m off to Austin for the Austin Workshop on Test Automation (awta). Every year I’ve wanted to attend this, but this is the first year I’ve made it. I was especially motivated because the subject is Watir.

We’ve been using Watir for about 4 years. We have a robust suite of tests and I think it was fairly well-designed (by a coworker who had Perl experience and decent OO understanding). We use these tests mainly to help with exploratory testing. They are very ‘smart’, they have a lot of logic to cope with whatever they encounter, and each script accepts many run-time parameters that make them quite flexible in what they test. This power comes at a cost – maintenance and updates can be time-consuming and difficult. Still, we could not do nearly as much exploratory testing without them (not to mention, we would go crazy from the tedium of manually creating lots of different scenarios for testing).

We’re a small team and we all have to multi-task to some extent. I spend the bulk of my time working with the customers and turning their examples and requirements into business-facing tests that help drive development, and doing manual exploratory testing on these. We automate 100% of our regression tests at the unit, behind-the-GUI (FitNesse) and GUI (Canoo WebTest and Watir), but automating doesn’t usually take that much of our time.

However, because we’re so busy, it’s hard to find big blocks of time to keep up with what’s new in each of the tools we use, upgrade to the latest version, and take advantage of new features. We’ve started to accumulate technical debt in our Watir scripts – we get sporadic failures when we run the full suite of tests, and haven’t been able to figure out why (all the tests run fine individually, and often run fine in the suite).

I recently spent a lot of time upgrading to the latest Watir version, getting it working on my new Vista, and on my Mac VMWare with Windows XP. It was a frustrating and painful experience and I got a lot of help from my team and from people in my Twitterverse. And I still can’t figure out why tests get sporadic failures in the suite.

At the same time, our FitNesse version is way behind, and we aren’t taking advantages of Slim and other new features of FitNesse. We’re overdue for an “engineering sprint” where we normally get a whole iteration to do things like upgrade our tools and learn new features. It’s hurting us. Recently we’ve had more failures of the build that runs our functional regression tests, and spent more time getting it green again. That’s time taken from the whole team, as everyone on the team addresses the problems.

So I’m thrilled to be heading off for three days of soaking up new information about how other people use Watir to address their test automation challenges, and hopefully get help with our Watir issues. I expect to have lots of ‘aha’ moments, and perhaps come home with a whole new approach for my team to try.

Even if your team, like ours, has been quite successful with test automation, don’t get complacent. Stay on top of the latest in both the tools you use and new tools that are out there. Refactor your tests and make sure they’re still meeting your needs. Don’t do like I have done lately and just give up running one of your regression suites because you don’t have time to diagnose why it’s having problems! (I’m atoning for that now). Budget time every iteration for the team to make sure your test automation is working for you.