Agile Testing with Lisa Crispin

I’ve submitted three proposals to Agile 2010 and would love your feedback. Please review them – you will need to create a free account on the submission site if you don’t already have one.

Workshop: Who Pays for All the Fun? Selling the Value of Testing in Agile Projects

Tutorial: The Whole Team Approach to Testing

Talk: How Low Can You Go: Doing the Defect Management Limbo

I have a thick skin, so don’t hold back. I know there will be lots of great sessions, there’s a high standard, and I want to make a useful contribution.

I wrote a blog post on StickyMinds about my views on the importance of being nice. I particularly like some of the stories that commenters have shared about how they were able to get a good working relationship with difficult coworkers. Please share your stories!

It’s that time of year where we may have a chance for extra socializing and pleasant distractions from the short, dark (at least in this hemisphere) days. Here’s a bit of holiday cheer from me, my husband Bob Downing, our dogs Tango and Bruno, and our donkeys Ernest and Chester.

Happy Holidays!

Beautiful Testing, with a wide variety of fascinating chapters contributed by leading software professionals, is available. Not only will it inspire you and your team to improve your testing and your product, but all royalties go to a good cause. I’m honored to have written a chapter. Please see the page on my site for more details and a free copy of my chapter – to whet your appetite for the rest of the book!

I meet more and more teams, both within my company and externally, who are getting really good at agile testing. They’ve adopted a Whole Team approach, they write tests first, they collaborate and communicate well, they use feedback to improve. Agile teams are functioning much better overall than what I was seeing a couple of years ago.

One perennial issue – and this isn’t new with agile – is that testers and teams still struggle with how to design and write automated tests that have a good return on investment. That is, the tests are reliable, stable, maintainable, and provide good value for the amount of work that goes into them. We also all struggle with how much to automate, what to not automate, what to keep in our regression suite.

Janet and I devoted a section of our book to automation, and we presented principles, techniques and strategies that will help. We also include this in our Agile Testing course, and we both do conference tutorials on how to succeed with automation in agile projects. I think in addition to that, a lot of people need some practical hands-on experience. They need to be guided through an example of automating tests for a story, then they need to try it for themselves.

I’m developing an internal class for people at my company, and we also have ideas for workshops and other ways for teams who have solved some automation problems and developed good practices to share with other teams. I just mind mapped the ideas we have so far. At Adam Goucher’s suggestion, I’m posting my first draft here.

It’s my hope that I could extend this to a more generic class that would be appropriate for a conference tutorial. I’ve talked with some folks about collaborating on something like this and kind of making it open source, with several of us who could teach it, like the Watir tutorial that Bret Pettichord, Charley Baker and others have taught. But, others better than I have tried to teach hands-on courses like this and run into issues. What tools would we use that can easily be installed on all participants’ laptops? What if the participants have different skill levels? So, I don’t know if this is really possible. I’m interested in ideas, or if anyone has done this successfully.

Actually I’m sure this has been done successfully by people like Elfriede Dustin who have been specializing in test automation for years – I need to get her book out and re-read it. I’m not trying to steal anyone’s idea or say that this is a new original idea I thought up. I just see a gap in many teams’ ability to create the ‘right’ automated regression tests.

So, here is the mind map, you’ll have to open it in a new window (I’m sure there’s a better way to do this but I have to catch a plane), feedback welcomed.

Last month I had the great pleasure to participate in Agile Testing Days. I did a tutorial on using the Agile Testing Quadrants for test planning, and a keynote on 7 key success factors for agile testing success. More importantly, I got to attend a lot of great sessions by leading agile testing practitioners.

There were so many good sessions and I’ll only mention a few here. Mark your calendars for next year’s Agile testing Days, Oct. 4 – 6.

My slogan after this conference comes from Dave Evans (and from Mike Scott – not sure which one of them coined the phrase but they both talked about it): Respect the Tests. More about that in a minute.

Elisabeth Hendrickson always inspires me, and her keynote didn’t let me down. I’ve heard her definition of agile before, but it bears repeating: Delivering value in the form of releasable software at frequent, regular intervals at a sustainable pace, while adapting ot the changing needs of business. She quoted Mary Poppendieck as saying, if you want to go very fast, you have to be very disciplined. Sustainable pace means that we keep technical debt down so that the code doesn’t impede us.

Elisabeth presented her four sources of technical risk: Ambiguity, assumptions, dependencies and capacity, and her 7 key testing practices: TDD, ATDD, Exploratory Testing, Automated System Tests, Collective Test Ownership, CI, and one more that I think is automated unit tests (I can’t read my notes!)

Elisabeth also quoted Tobias Mayer, who refers to the Product Owner as the Voice of the What, and the dev team as the Tribe of the How. I like this, but I take issue with it a bit. The Poppendiecks have influenced me to experiment with having the dev team learn the business inside out, so the dev team itself can help the business prioritize and even think of features. This worked well for my team at ePlan, and I have to agree that funneling all the business requirements through the Product Owner has a lot of drawbacks.

What was new to me in this talk was something that several of us had talked about a night or two before, the idea of Collective Test Ownership. This ties in with the Whole Team Approach. Everyone on the team takes responsibility for quality, testing, and test automation.

Another great session, again from someone whose work I’m familiar with, was David Evans‘ talk. One big takeaway for me was this: If a test is worth writing, it’s worth automating, and it must always pass in the future. I often see teams that ignore failures in their CI tests because “We have to get these new stories out, this is the PO’s priority.” So the PO doesn’t care whether what she prioritized in previous sprints still works or not? When the CI breaks, the team should stop the line and get it green.

Dave pointed out that if all tests pass, you don’t need extra metrics. Defects are evidence of missing tests. So don’t call it a defect – call it a failing test, write the failing test and fix it right away. If the customer really doesn’t want it fixed now, write a story for later.

Dave recommend fixing time and quality, but don’t fix functional scope. If time is running out, thin the requirements. It’s not like leaving a wheel off or leaving out the engine. Respect the tests – any test worth writing is worth running all the time.

Dave presented his Agile Quality Framework. I won’t go into all the details, but one of my favorite parts is applying balloon patterns. An empty balloon is still a balloon, but if you don’t have the rubber outside, it’s hard to have the air and then add the rubber later. Do the rubber first – the CI, the test framework.

Mike Scott elaborated on the balloon pattern in his demo of Testify Wizard. In one hour, Mike went from zero to a full CI process, including the project two continuous builds, one running unit tests, and one running FitNesse tests. It even generated code coverage reports! I think he did this with both Java and .Net during the same session. He did this with a couple of pushes of the button,some minor configuration and a wave of his crutch.

This shows how easy it is to respect the tests. It’s not that hard to build a framework to integrate our code and run our automated tests so that we get quick feedback all the time.

I’m doing a tutorial and a keynote at Agile Testing Days in Berlin. I think this is the first big agile testing conference to be held in Germany. Take a look at the program – it’s an awesome lineup of agilists. And since the most helpful part of conferences is always the informal connections you make, I’m happy to say that I know of a lot of interesting people planning to participate also.

Here are a couple of discount codes:

10% for the Tutorials: code SPEAKERSTUT010

20% for the conference: Code SPEAKERCON020

I hope to see you there!

If you’re in the Oslo area, don’t forget I am doing an Agile Testing course at Program Utvikling the following week.

There are a lot of creative, smart people on our team! Unfortunately I missed the discussion where these ideas come out, but we are trying some ways to organize such a big team. We will have “cells” or sub-teams, each one will take on a set of stories.

Today, each cell wrote high level acceptance tests together. Our cell started out doing BDD style tests on a FitNesse wiki. We used my desktop and the group in Weston VNC’d in and projected it on screen. But they couldn’t scroll up and down. And tests were taking too long to write. In our pomodoro retro, we decided to do high level test cases together as bullet points only, then have a pair flesh the tests out into BDD style and automate them.

We also decided to try Google docs. We kept using VNC and projecting the screen, but since the page is saved often, each person could also look at the doc on his own machine and scroll up and down. Both changes worked better. I had to split off for a long phone call with someone from an internal development team (that’s a whole ‘nother blog post), and when I came back they had finished the tests.

We are doing what I just learned is set-based engineering to decide whether we’d rather write our SWAT tests in C# and Visual Studio, or in FitNesse. We will do both this sprint and see which is better. My teammate Maykel started writing FitNesse fixtures that would allow using BDD tests to run SWAT from FitNesse, and I think he also had some way to make the tests run faster in FitNesse. One pair will automate the acceptance tests, while the other pair does a spike.

Thanks to the people who commented on yesterday’s post, I feel much better about having the testers get together occasionally. It’s true that we should have a testing community within our agile team, just as we have started one company-wide. I think we are on the right path with this team, despite it being larger than I would like.

I’ve worked on agile teams where programmers, testers, DBAs, sys admins and other roles all saw themselves as part of one developer team for so long, the idea of a separate test team gives me a chill. Yet even on experienced agile teams, there are still testers who see themselves as a “test team”.

I need to investigate more what they mean by this, I might be misinterpreting it. My new team is the combination of three teams, totalling 17 people (not counting the business folks). I think that’s unwieldy. Today, an experienced tester on my new team, who has worked on an agile team for some period of time (I’m not sure how long), proposed that the “test team” get together and discuss practices and strategy. I about blew a gasket, but that didn’t keep the meeting from happening (I’m always on phone and video with my team, so I can’t hep being in the meetings.)

The content of the meeting was reasonable enough. One topic was patterns of tests that we do, for example, for a numeric field you would always try certain test cases such as alpha and special characters, null, etc. Standard stuff.

The next topic was about automating tests on a page and end-to-end level, in addition to at the story level. This bugged me. If you tie tests to a page in the UI, and later stuff gets moved around so that what used to be on that one page is now on different pages, your tests will break. As far as automating end-to-end tests. we definitely have to be careful since the ROI can be bad on automating those.

My teammate Chris McMahon explained how our team up to now has been trying to group tests at the domain level. One FitNesse test contains all tests related to a rating scale, for example, until there are about 200 test steps and then we split it up into some reasonable sub-grouping. We’re testing at a feature level, no matter where we are in the UI.

The next topic was localization. Our app accommodates data in three languages. Someone proposed that we design our tests so that they could be run for any language. This sounds good to me, but how hard is it to do, and is it worth it? Is the localization already tested by the team that does the framework for it? I suggested we definitely needed programmer agreement for this topic, and everyone agreed.

I then brought up topics I’ve been wanting to raise, the non-functional testing. Our company has teams to do performance, scalability, reliability and security testing, but isn’t our team still responsible to make sure those happen in a timely manner? This discussion proved helpful for me, as I learned about the teams that do this testing, and how we can become involved and do a lot of it ourselves. I’m glad to get these activities out in the open and on our minds so they won’t be overlooked.

In the end I needn’t have been so spooked by the idea of testers meeting together. As it turned out, the programmers got into a design discussion with the database guy, and we were not involved in that (welll, I didn’t know about it until it was underway), so it was an opportunity for the testers to talk. But I don’t want that to happen too often.

I’m looking forward to starting work on stories. I think our test design will evolve as we write new code. We will pair on everything, so I hope that means the programmers and testers will remain engaged with each other.

It’s so cool to know that Janet and I have communicated clearly enough what we hoped everyone would take away from our book. Paul Grenyer’s review is a thrill for me.

My teammates Maykel Suarez, J.P. Erkelens and  Eddy Lara ought to be the ones blogging about this as they come up with these cool ideas, but I have the time and the inclination so I will. I hope they’ll read this and correct any errors I make.

I’ve used FitNesse since 2003 and obviously have not learned enough up to now!

We want our FitNesse tests to be DRY (Don’t repeat yourself), easy to maintain and easy to read and understand. Maykel and I were pairing on some FitNesse/SWAT tests this morning and found we needed the same test steps in more than one place. Maykel created this page under our “HelperMacro” area. Each FitNesse variable contains a SWAT test table.

!define NavigateToRatingScale (
!|SWAT |
|StimulateElement|Expression|