Agile Testing with Lisa Crispin

I appreciated David Evans’ review of Gojko Adzic‘s new book, Bridging the Communication Gap: Specification by Example and Agile Acceptance Testing. David makes the point that agile quality is a whole-team responsibility, and that Gojko changed his original title from “Agile Acceptance Testing” so that anyone on an agile team, not just testers, would read it.

Janet and I push the Whole Team approach throughout our book as well. In fact, it’s #1 on our list of “key success factors” for agile testing. No team will succeed in the long run unless everyone on the team is committed to delivering the highest possible quality. We all know it’s impossible to “test quality in”. Tests form the foundation of our ability to delight our customers. But it’s so easy to misunderstand or miss requirements, even if you’re a disciplined team following all the agile principles and practices.

Here’s my latest concrete example of the whole team approach. My PC got upgraded to Vista 64 bit, and at the same time we upgraded to the latest Watir version (1.6.2). It took me a lot of time to get the Watir suites working again, not only due to the upgrades but because of technical debt. We had let a lot of little problems in the scripts slide, and failed to update them for all the changes in the app.

Who all was involved in this effort?

• The product owner agreed to a two-point story to get the scripts working again. (The story got put off for two sprints due to other emergencies, but that was partly my decision). So the business agreed to the importance of knocking out this technical debt, getting back the ability to run some important regression tests, and maintaining the scripts’ ability to speed up exploratory testing.
  
• A teammate who is a sys admin and programmer helped me with Vista settings (if my hard drive goes to sleep, the suite stops and waits) and differences in IE behavior (whether a new window is started or the old one used, having to confirm close window, etc)
• A teammate who is a tester and programmer paired with me some to fix bugs and problems, for example, regular expressions that didn’t work anymore because the html changed.
• When I learned from other Watir users that modal dialogs are old-school, and there are newer and better techniques, I discussed this with the other programmers who agreed. This might start an effort to evaluate whether we should change these in the interest of testability.

This example doesn’t speak so much to Gojko’s communication solutions, since it doesn’t deal with coding new functionality, but it does show how the whole team gets involved in anything to do with testing, and with keeping our app testable.

If you don’t currently enjoy working with a team whose members are all committed to quality and testing, read both of our books to learn why you need this whole team commitment, and some ways you might get it.

Weekend before last I had the great pleasure of participating in the Austin Workshop for Test Automation. Test tool developers and users exchanged experiences and ideas. You know that automated testing has come a long way when you see the approaches organizations such as Opera and Expedia use to overcome massive testing challenges.

One discussion that helped a penny drop in my brain was the concept of test libraries vs. test frameworks. A library is a reusable component. Watir is a family of Ruby libraries that drive the browser. A framework is a set of libraries that gives you everything you need to test. Cucumber is a test framework that supports behavior-driven development using plain text. Taza is a test framework that uses the Watir libraries and generates RSpec test assertions. At least – I think so! Please correct me if I’m wrong.

A good analogy someone came up with at AWTA is that Kevin O’Connor’s (host of This Old House) tool shop is a framework – it has everything needed for any project, whereas Marek J‘s garage is just a bunch of tools. This is not to cast aspersion on Marek J’s own cool test automation framework, Watirloo.

What I learned is that my team has our own test automation framework using Watir, which consists of our Ruby scripts and test/unit. The BDD frameworks such as RSpec and Cucumber seem to be the cooler thing to use than test/unit, but I don’t know how we would take on the huge task of refactoring all our Ruby/Watir scripts to use a different test framework.

That doesn’t mean we shouldn’t look at some of these newer frameworks to see if they have something to offer that our current automation solutions don’t include. My fellow tester Lori Bergan has been looking into Cucumber. I’ve been intrigued by BDD since last year’s CITCON Denver. I like tools that seem natural.

What tools have you been trying lately? Are you happy with the ones you currently use?

Check out the Agile 2009 website for information on how to propose sessions. Bob Galen and I, along with a stellar review committee, are looking for practical sessions for all levels of experience to include on the Testing Stage. Whether you’re doing a set of practices such as Scrum, XP, Crystal, or your own flavor of agile, we hope you’ll give back to the agile community by sharing your experiences and ideas. There are lots of interesting stages, in fact you might have a hard time deciding where your agile testing or other topic session belongs. Don’t sweat it, you can work with the stage producers to decide where it fits best.

By submitting now, you’ll have a couple of weeks in which reviewers will give you plenty of feedback, ask questions to help you clarify your proposal, and polish it up. Please submit today!

I got positive feedback on this post to the agile-testing Yahoogroup (including from Brian Marick), so I decided to post it here too for anyone who missed it on the list.

My team took time to educate the business management about technical debt and how it affects our velocity. They can see our velocity start to go down if we accumulate too much technical debt. This means fewer new features. We’ve tried to translate technical debt into things they can understand – dollars and cents, less ability to deliver features in a timely manner.

Our agreement with the business is that we get one two-week iteration every 6 months for ourselves, to do big refactorings (we always refactor, but sometimes it’s risky to do in a regular iteration), to upgrade our tools, to try out new tools. For example, we switched from CVS to SVN during a “refactoring” sprint. If a major refactoring is needed in the production code, we do it early in the “refactoring” iteration so we have lots of time to test it.

Over the years the business has seen the benefits of our “refactoring” sprints. They see the velocity go back up and how the investment of a small amount of time pays off.

People say, “My business will never let us do that”. You have to build trust between the software team and the business team. My team puts a lot of effort into understanding the business. We sit with different people and watch how they work. We document business processes and suggest ways we could help with software. We can help the business by suggesting a lower-cost or higher-ROI solution to a business problem. We know enough about the business to help set priorities. When a new theme is coming up, we can identify dependencies and help find the most efficient way to develop the theme. We identify “bling” and help keep the business looking at ROI. We’re all invested in a successful business, after all.

We’re currently overdue for a “refactoring” sprint, so we’ve had to start writing stories and budgeting extra time for activities to reduce our technical debt in each sprint. The business people know they are getting lower velocity, it’s their choice to put off the refactoring sprint and they know they’re paying a price for it.

Your team needs to find what works for you, but try to open a dialog with the business managers, try to understand them and help them understand what technical debt is and how it hurts the business bottom line.

Tomorrow I’m off to Austin for the Austin Workshop on Test Automation (awta). Every year I’ve wanted to attend this, but this is the first year I’ve made it. I was especially motivated because the subject is Watir.

We’ve been using Watir for about 4 years. We have a robust suite of tests and I think it was fairly well-designed (by a coworker who had Perl experience and decent OO understanding). We use these tests mainly to help with exploratory testing. They are very ‘smart’, they have a lot of logic to cope with whatever they encounter, and each script accepts many run-time parameters that make them quite flexible in what they test. This power comes at a cost – maintenance and updates can be time-consuming and difficult. Still, we could not do nearly as much exploratory testing without them (not to mention, we would go crazy from the tedium of manually creating lots of different scenarios for testing).

We’re a small team and we all have to multi-task to some extent. I spend the bulk of my time working with the customers and turning their examples and requirements into business-facing tests that help drive development, and doing manual exploratory testing on these. We automate 100% of our regression tests at the unit, behind-the-GUI (FitNesse) and GUI (Canoo WebTest and Watir), but automating doesn’t usually take that much of our time.

However, because we’re so busy, it’s hard to find big blocks of time to keep up with what’s new in each of the tools we use, upgrade to the latest version, and take advantage of new features. We’ve started to accumulate technical debt in our Watir scripts – we get sporadic failures when we run the full suite of tests, and haven’t been able to figure out why (all the tests run fine individually, and often run fine in the suite).

I recently spent a lot of time upgrading to the latest Watir version, getting it working on my new Vista, and on my Mac VMWare with Windows XP. It was a frustrating and painful experience and I got a lot of help from my team and from people in my Twitterverse. And I still can’t figure out why tests get sporadic failures in the suite.

At the same time, our FitNesse version is way behind, and we aren’t taking advantages of Slim and other new features of FitNesse. We’re overdue for an “engineering sprint” where we normally get a whole iteration to do things like upgrade our tools and learn new features. It’s hurting us. Recently we’ve had more failures of the build that runs our functional regression tests, and spent more time getting it green again. That’s time taken from the whole team, as everyone on the team addresses the problems.

So I’m thrilled to be heading off for three days of soaking up new information about how other people use Watir to address their test automation challenges, and hopefully get help with our Watir issues. I expect to have lots of ‘aha’ moments, and perhaps come home with a whole new approach for my team to try.

Even if your team, like ours, has been quite successful with test automation, don’t get complacent. Stay on top of the latest in both the tools you use and new tools that are out there. Refactor your tests and make sure they’re still meeting your needs. Don’t do like I have done lately and just give up running one of your regression suites because you don’t have time to diagnose why it’s having problems! (I’m atoning for that now). Budget time every iteration for the team to make sure your test automation is working for you.

I had the good fortune to be in Salt Lake City last Friday and attend the last 1.5 hours of the Salt Lake City Agile Round Table. Participants included Alistair Cockburn, Kay Johansen, Zhon Johansen, Jeff Patton, Jonathan House, Sean Landis, John Major and other smart people (sorry, I am bad with names!) I got lots of “ahas” and food for thought. You can see Sean Landis’ notes for some of the interesting discussion on traps & pitfalls of agile. (Their mailing list is sl-agile in Yahoogroups).

One topic discussed was that given that we all agree that agile adoption isn’t process change, it’s culture change, why do we teach process rather than culture and values? I think the reason (and most participants said this) is that teaching process is a lot easier than trying to teach values. We’re used to learning in a straightforward “how to” way – follow these steps and you will succeed. But that doesn’t work, because everyone has a unique situation.

What values do you need to succeed as an agile tester, or with testing on an agile team? In our book, we identify 7 key success factors for agile testing. Are these values or process? I think they are values:

• Use the whole-team approach. The whole development team must commit to producing high-quality software that delivers maximum business value.
• Adopt an agile testing mind-set. We have a chapter in the book with “Ten Principles for Agile Testers”, and explain that an agile testing attitude is proactive, creative, open to new ideas, and willing to take on any task.
• Automate regression testing. Without automated regression tests, the team will fail in the long run, because there won’t be time to do adequate exploratory testing, much less to do all the manual regression tests as the product grows. This is really a practice, but its root is in the value of feedback.
• And feedback is our next success factor: Provide and Obtain Feedback. Feedback is a core agile value, and testers are in a prime position to help provide the feedback that keeps the team on course.
• Build a foundation of core practices. Again, these are practices such as continuous integration and working incrementally, but they are rooted in core agile values such as simplicity, communication and feedback.
• Collaborate with customers – another core agile value.
• Look at the big picture – another strength of testers. Even code produced test-first may cause unintended results in other parts of the system. Testers help the team evaluate how each story fits into the larger system.

I’m happy that our book focuses a lot on cultural issues, agile values and principles and how they relate to testing on agile teams. As I worked over the weekend on my presentations for TISQA, I found that I also tend to teach process rather than values. I updated my presentations to make sure I am explaining values, principles and the importance of letting those guide you, rather than try to provide a cookbook approach that probably won’t work for everyone.

Do you agree that the success factors are values, or am I deluding myself?