Agile Testing with Lisa Crispin

This month’s issue Dressage Today had some articles about the two-time Olympian dressage rider and trainer, Steffen Peters. He talked about the role of discipline in turning dreams into reality. For him, this means keeping himself mentally and physically fit through a thoughtfully planned exercise and diet routine, as well as understanding the capabilities of each horse and how it can learn best.

My own dressage trainer tells me all the time, “It takes a million repetitions to change a bad habit.” That’s only a slight exaggeration in my case. I remind myself (and she reminds me) about 50 times a ride to correct each bad habit, for example, to put my hands closer together and keep them centered over the horse’s withers. Some of my bad habits are 50 years old, so they’re hard to fix, but I’m slowly getting better.

Discipline is a critical component of delivering high-quality software, too. We need to thoughtfully plan to use the right practices for us. We need to remind ourselves every day, via big visible charts, information radiators, retrospectives, any means that works to learn new good habits and reinforce the things we do well.

During a recent estimating meeting, our product owner (who normally knows better) asked, “About 6 weeks ago you estimated this XYZ story at 5 points. Is there any cheap, hacky way to do it for one point?” One of the developers responded, “Possibly, but we are not going to do a cheap, hacky solution”. It’s tempting to cut corners to meet a business need, but hacking in a quick fix that increases our technical debt down the road isn’t the way to do it.

Another example where discipline and good habits came into play occurred in this morning’s Scrum. A couple of the customers brought up a significant piece functionality that they had accidentally omitted from a story we’re already working on. It was tempting to say “OK, we’ll add that in too.” But it was too big a change to put into the story at this late date. We discussed possible workarounds, and one of the customers came up with one that would do the job. They will write a new story, and if we can, we’ll pull it in this sprint. If not, they’ll get it two weeks later.

Sometimes we do make compromises about quality, and implement a solution that is less than ideal. There are always business trade-offs to consider. But we don’t make these lightly. And even after more than five years of agile development, we spend time on a serious retrospective every iteration, and work hard to keep improving the way we work and the quality of our code. Some of our bad habits have been hard to fix, but we keep working at it. Our business continues to grow, which to me is the best feedback about our disciplined approach to keeping up good habits and eliminating bad habits.

Janet Gregory and I had the pleasure of participating in a discussion related to our book this week on Java Ranch in the Big Moose Saloon forum. There were some really good questions and resulting discussions. Take a look.

An article by Janet and me just got posted on InformIT – “Testers: The Hidden Resource“. This was an idea that popped in my head one day when I was driving to work, after we had just finished the book. Testers have a lot more to offer than most development organizations realize. See what you think, I’d love to hear your comments. Are we too optimistic or do you agree?

Phil Kirkham had the interesting idea to ask authors of different testing blogs the same five questions about blogging. I was honored to be asked, and Phil has posted my responses to 5 Questions for Lisa Crispin. I especially liked the photo he used. (I have more donkey photos on my old website.) It was interesting to read Matt Heuser‘s responses to the same questions, and I look forward to the responses from other testing bloggers.

I have a lot of links to testing blogs now in my Favorite Links section. Am I missing some good ones? Please let me know, I’m sure I must be.

Zeliko Filipin has posted a bunch of podcasts with AWTA workshop participants, including yours truly and Janet Gregory. I knew at the time that I wasn’t grasping all the information in every presentation, and listening to the podcasts reminds me of what didn’t quite register at the time. There are so many possibilities for test automation, and I need to make sure I learn about as many new frameworks and tools as I can, as more and better ones become available.

I invested a lot of time over the past few weeks upgrading to the latest version of Watir and getting the scripts and suites working in Vista. The silver lining was getting a bit familiar again with Ruby. We don’t do a lot of new stuff with these scripts, so my skills (such as they are) were rusty. It’s personal technical debt to let that happen, on top of the technical debt of scripts that weren’t updated to work properly with new code. But now my technical debt burden is lighter.

Next on my personal task board is to upgrade to the latest FitNesse. We’re on a very old version, and I know Uncle Bob Martin has made a lot of improvements lately, including the ability to use Slim instead of Fit underneath. I get overwhelmed trying to keep up with tools (someone asked me at AWTA about the overhead of using multiple test tools, and it’s true) but I have to invest time in that. The ROI demands it.

I appreciated David Evans’ review of Gojko Adzic‘s new book, Bridging the Communication Gap: Specification by Example and Agile Acceptance Testing. David makes the point that agile quality is a whole-team responsibility, and that Gojko changed his original title from “Agile Acceptance Testing” so that anyone on an agile team, not just testers, would read it.

Janet and I push the Whole Team approach throughout our book as well. In fact, it’s #1 on our list of “key success factors” for agile testing. No team will succeed in the long run unless everyone on the team is committed to delivering the highest possible quality. We all know it’s impossible to “test quality in”. Tests form the foundation of our ability to delight our customers. But it’s so easy to misunderstand or miss requirements, even if you’re a disciplined team following all the agile principles and practices.

Here’s my latest concrete example of the whole team approach. My PC got upgraded to Vista 64 bit, and at the same time we upgraded to the latest Watir version (1.6.2). It took me a lot of time to get the Watir suites working again, not only due to the upgrades but because of technical debt. We had let a lot of little problems in the scripts slide, and failed to update them for all the changes in the app.

Who all was involved in this effort?

• The product owner agreed to a two-point story to get the scripts working again. (The story got put off for two sprints due to other emergencies, but that was partly my decision). So the business agreed to the importance of knocking out this technical debt, getting back the ability to run some important regression tests, and maintaining the scripts’ ability to speed up exploratory testing.
  
• A teammate who is a sys admin and programmer helped me with Vista settings (if my hard drive goes to sleep, the suite stops and waits) and differences in IE behavior (whether a new window is started or the old one used, having to confirm close window, etc)
• A teammate who is a tester and programmer paired with me some to fix bugs and problems, for example, regular expressions that didn’t work anymore because the html changed.
• When I learned from other Watir users that modal dialogs are old-school, and there are newer and better techniques, I discussed this with the other programmers who agreed. This might start an effort to evaluate whether we should change these in the interest of testability.

This example doesn’t speak so much to Gojko’s communication solutions, since it doesn’t deal with coding new functionality, but it does show how the whole team gets involved in anything to do with testing, and with keeping our app testable.

If you don’t currently enjoy working with a team whose members are all committed to quality and testing, read both of our books to learn why you need this whole team commitment, and some ways you might get it.

Weekend before last I had the great pleasure of participating in the Austin Workshop for Test Automation. Test tool developers and users exchanged experiences and ideas. You know that automated testing has come a long way when you see the approaches organizations such as Opera and Expedia use to overcome massive testing challenges.

One discussion that helped a penny drop in my brain was the concept of test libraries vs. test frameworks. A library is a reusable component. Watir is a family of Ruby libraries that drive the browser. A framework is a set of libraries that gives you everything you need to test. Cucumber is a test framework that supports behavior-driven development using plain text. Taza is a test framework that uses the Watir libraries and generates RSpec test assertions. At least – I think so! Please correct me if I’m wrong.

A good analogy someone came up with at AWTA is that Kevin O’Connor’s (host of This Old House) tool shop is a framework – it has everything needed for any project, whereas Marek J‘s garage is just a bunch of tools. This is not to cast aspersion on Marek J’s own cool test automation framework, Watirloo.

What I learned is that my team has our own test automation framework using Watir, which consists of our Ruby scripts and test/unit. The BDD frameworks such as RSpec and Cucumber seem to be the cooler thing to use than test/unit, but I don’t know how we would take on the huge task of refactoring all our Ruby/Watir scripts to use a different test framework.

That doesn’t mean we shouldn’t look at some of these newer frameworks to see if they have something to offer that our current automation solutions don’t include. My fellow tester Lori Bergan has been looking into Cucumber. I’ve been intrigued by BDD since last year’s CITCON Denver. I like tools that seem natural.

What tools have you been trying lately? Are you happy with the ones you currently use?

Check out the Agile 2009 website for information on how to propose sessions. Bob Galen and I, along with a stellar review committee, are looking for practical sessions for all levels of experience to include on the Testing Stage. Whether you’re doing a set of practices such as Scrum, XP, Crystal, or your own flavor of agile, we hope you’ll give back to the agile community by sharing your experiences and ideas. There are lots of interesting stages, in fact you might have a hard time deciding where your agile testing or other topic session belongs. Don’t sweat it, you can work with the stage producers to decide where it fits best.

By submitting now, you’ll have a couple of weeks in which reviewers will give you plenty of feedback, ask questions to help you clarify your proposal, and polish it up. Please submit today!

I got positive feedback on this post to the agile-testing Yahoogroup (including from Brian Marick), so I decided to post it here too for anyone who missed it on the list.

My team took time to educate the business management about technical debt and how it affects our velocity. They can see our velocity start to go down if we accumulate too much technical debt. This means fewer new features. We’ve tried to translate technical debt into things they can understand – dollars and cents, less ability to deliver features in a timely manner.

Our agreement with the business is that we get one two-week iteration every 6 months for ourselves, to do big refactorings (we always refactor, but sometimes it’s risky to do in a regular iteration), to upgrade our tools, to try out new tools. For example, we switched from CVS to SVN during a “refactoring” sprint. If a major refactoring is needed in the production code, we do it early in the “refactoring” iteration so we have lots of time to test it.

Over the years the business has seen the benefits of our “refactoring” sprints. They see the velocity go back up and how the investment of a small amount of time pays off.

People say, “My business will never let us do that”. You have to build trust between the software team and the business team. My team puts a lot of effort into understanding the business. We sit with different people and watch how they work. We document business processes and suggest ways we could help with software. We can help the business by suggesting a lower-cost or higher-ROI solution to a business problem. We know enough about the business to help set priorities. When a new theme is coming up, we can identify dependencies and help find the most efficient way to develop the theme. We identify “bling” and help keep the business looking at ROI. We’re all invested in a successful business, after all.

We’re currently overdue for a “refactoring” sprint, so we’ve had to start writing stories and budgeting extra time for activities to reduce our technical debt in each sprint. The business people know they are getting lower velocity, it’s their choice to put off the refactoring sprint and they know they’re paying a price for it.

Your team needs to find what works for you, but try to open a dialog with the business managers, try to understand them and help them understand what technical debt is and how it hurts the business bottom line.

Tomorrow I’m off to Austin for the Austin Workshop on Test Automation (awta). Every year I’ve wanted to attend this, but this is the first year I’ve made it. I was especially motivated because the subject is Watir.

We’ve been using Watir for about 4 years. We have a robust suite of tests and I think it was fairly well-designed (by a coworker who had Perl experience and decent OO understanding). We use these tests mainly to help with exploratory testing. They are very ‘smart’, they have a lot of logic to cope with whatever they encounter, and each script accepts many run-time parameters that make them quite flexible in what they test. This power comes at a cost – maintenance and updates can be time-consuming and difficult. Still, we could not do nearly as much exploratory testing without them (not to mention, we would go crazy from the tedium of manually creating lots of different scenarios for testing).

We’re a small team and we all have to multi-task to some extent. I spend the bulk of my time working with the customers and turning their examples and requirements into business-facing tests that help drive development, and doing manual exploratory testing on these. We automate 100% of our regression tests at the unit, behind-the-GUI (FitNesse) and GUI (Canoo WebTest and Watir), but automating doesn’t usually take that much of our time.

However, because we’re so busy, it’s hard to find big blocks of time to keep up with what’s new in each of the tools we use, upgrade to the latest version, and take advantage of new features. We’ve started to accumulate technical debt in our Watir scripts – we get sporadic failures when we run the full suite of tests, and haven’t been able to figure out why (all the tests run fine individually, and often run fine in the suite).

I recently spent a lot of time upgrading to the latest Watir version, getting it working on my new Vista, and on my Mac VMWare with Windows XP. It was a frustrating and painful experience and I got a lot of help from my team and from people in my Twitterverse. And I still can’t figure out why tests get sporadic failures in the suite.

At the same time, our FitNesse version is way behind, and we aren’t taking advantages of Slim and other new features of FitNesse. We’re overdue for an “engineering sprint” where we normally get a whole iteration to do things like upgrade our tools and learn new features. It’s hurting us. Recently we’ve had more failures of the build that runs our functional regression tests, and spent more time getting it green again. That’s time taken from the whole team, as everyone on the team addresses the problems.

So I’m thrilled to be heading off for three days of soaking up new information about how other people use Watir to address their test automation challenges, and hopefully get help with our Watir issues. I expect to have lots of ‘aha’ moments, and perhaps come home with a whole new approach for my team to try.

Even if your team, like ours, has been quite successful with test automation, don’t get complacent. Stay on top of the latest in both the tools you use and new tools that are out there. Refactor your tests and make sure they’re still meeting your needs. Don’t do like I have done lately and just give up running one of your regression suites because you don’t have time to diagnose why it’s having problems! (I’m atoning for that now). Budget time every iteration for the team to make sure your test automation is working for you.