Agile Testing with Lisa Crispin

My team was lucky in having Brian Marick visit us last week, interviewing some of us for his book “Travels in Software”. He had visited us back in 2004, and I hope it was interesting for him to see that 1) we used a lot of ideas we got from him, such as using examples to drive coding and 2) we’ve changed and grown a lot and are a productive team.

It was fun to be interviewed by Brian, and he asked thought-provoking questions. One question was, what would you go back and tell your 2003 self, when you first started on this team? Or presumably, even my 2000 self on my first agile team. This has started a couple different thoughts in my head.

Here’s one thing I’d be sure to tell myself. “Lisa, you are good at picking up a new domain, and you’ll learn about 401(k) plans, all their government regulations, how all the different users use the application and other knowledge directly related to the application under test pretty fast. However, there’s lots more to know than you think. You’ll still be learning about parts of the app you never saw before in 2009! Also, you’d better learn about the whole business, not just the parts that are automated in the app.”

Since Day 1 of implementing Scrum, our team has been committed to delivering the highest possible quality software, and I think we’ve done a fairly good job. However, one of our biggest problems has been that we sometimes misunderstand customer requirements, or miss requirements entirely, even though we are driving development with tests.

It was only after we made a conscious, concerted effort to sit with the business people, learn all their processes in detail, and document them, that we got a full enough understanding of the business. For example, only recently did I learn about all the different bank accounts through which money moves around, because much of the movement is done manually. It’s critical to know this, however, because it’s very bad when cash balances stop reconciling correctly, or when the clearinghouse or bank says we have a different amount in each account than we think we do.

Maybe it was important that we first master agile practices such as TDD, CI, refactoring, driving coding with examples and tests at all levels, and test automation, before we moved on to getting a broad and deep understanding of the business and of every person’s job. You can’t do everything at once. Still, it would have helped me to at least have a sense of what I didn’t know yet, so I didn’t have a false sense of confidence.

Do you know your business inside and out? Or only the application for which your team writes code?

Chapter 21, Key Success Factors, is available on InfoQ.

Please read and comment! I look forward to your feedback.

Jeff Langr and Tim Ottinger have added a flash card to their “Agile: In a Flash” project (that will become a book) based on our book’s “Ten Principles for Agile Testers”. They did a nice job of summarizing the information in Chapter 2 of Agile Testing: A Practical Guide for Testers and Agile Teams. We are honored to have this information as part of the terrific Agile flash card project!

When Brian Marick came up with his ideas about using examples to drive development (I think that was back in 2003), my team latched on to the concept. We are fortunate in having an expert product owner who is talented at coming up with good examples. People often ask me how this works. Our current and previous sprints contain a good illustration of how we use examples to drive testing and coding.

We’re working on a theme to display internal rate of return on account statements. This involves a complex algorithm. Here’s just one part of the formula. Our development team hashed out this formula together with the product owner in the course of a couple of meetings to discuss it. Because our programmers and testers have a lot of domain knowledge, we were able to question the PO’s initial formula, and come up with one that was simpler to code and still provided a correct value.

The reason we were doing this theme is that our previous IRR formula got way off with edge cases, for example, if a participant started out a quarter with a zero balance, and got a big asset rollover in the middle of the quarter. The product owner researched nine different edge cases, pulled their data out of the production database, and created spreadsheets showing all the cash flows and the resulting IRR value. Here’s a small snippet of one of the spreadsheets, showing some of the cash flows for the first quarter of 2008, and the IRR value.

A developer and tester worked together to first write a simple FitNesse test (here’s a snippet of the FitNesse test) using data from one of the product owner’s examples. The IRRs came out different than the product owner’s. The PO did some research and found some problems with his own calculations. This process iterated a few times until both the example in the spreadsheet and the one in the FitNesse test produced the correct values.

From there, the tester has been creating more FitNesse tests using the other examples from the spreadsheet. All the known edge cases are in the spreadsheet, so this is a great start on covering all the possible test cases. We’ll do more tests, both through FitNesse, and through the UI, until we’re satisfied that the correct value displays on the account statements.

It’s taking us two sprints to complete the different stories in this theme. I think it would take much longer if we didn’t have the examples. The PO has saved us lots of time by researching the examples from production data up front, and putting those in a spreadsheet so we can see how he reached his IRR values. He worked closely with programmers and testers as the tests and code were developed in several tiny iterations until everyone was satisfied that the code was performing correctly, at least initially. Now we have his edge cases as examples to make sure we cover all possibilities, along with all the test cases we think of ourselves.

We have good confidence that we can deliver the right business value. As we have a firm deadline, this is a good thing. Some of the FitNesse tests will be put into our regression test suite that runs in a continuous build process. There are so many benefits to using examples to drive development.

Do you use examples to drive testing and coding? Please share your stories.

I am extremely proud that our book is #28 on Jurgen Koop’s list of the top 50 new software development books! Also flattered to be in such excellent company on this list. There are some titles there that I hadn’t heard about before – time to go order some books! Check out the list and see what’s missing from your pile of books to read.