Agile Testing with Lisa Crispin

I meet more and more teams, both within my company and externally, who are getting really good at agile testing. They’ve adopted a Whole Team approach, they write tests first, they collaborate and communicate well, they use feedback to improve. Agile teams are functioning much better overall than what I was seeing a couple of years ago.

One perennial issue – and this isn’t new with agile – is that testers and teams still struggle with how to design and write automated tests that have a good return on investment. That is, the tests are reliable, stable, maintainable, and provide good value for the amount of work that goes into them. We also all struggle with how much to automate, what to not automate, what to keep in our regression suite.

Janet and I devoted a section of our book to automation, and we presented principles, techniques and strategies that will help. We also include this in our Agile Testing course, and we both do conference tutorials on how to succeed with automation in agile projects. I think in addition to that, a lot of people need some practical hands-on experience. They need to be guided through an example of automating tests for a story, then they need to try it for themselves.

I’m developing an internal class for people at my company, and we also have ideas for workshops and other ways for teams who have solved some automation problems and developed good practices to share with other teams. I just mind mapped the ideas we have so far. At Adam Goucher’s suggestion, I’m posting my first draft here.

It’s my hope that I could extend this to a more generic class that would be appropriate for a conference tutorial. I’ve talked with some folks about collaborating on something like this and kind of making it open source, with several of us who could teach it, like the Watir tutorial that Bret Pettichord, Charley Baker and others have taught. But, others better than I have tried to teach hands-on courses like this and run into issues. What tools would we use that can easily be installed on all participants’ laptops? What if the participants have different skill levels? So, I don’t know if this is really possible. I’m interested in ideas, or if anyone has done this successfully.

Actually I’m sure this has been done successfully by people like Elfriede Dustin who have been specializing in test automation for years – I need to get her book out and re-read it. I’m not trying to steal anyone’s idea or say that this is a new original idea I thought up. I just see a gap in many teams’ ability to create the ‘right’ automated regression tests.

So, here is the mind map, you’ll have to open it in a new window (I’m sure there’s a better way to do this but I have to catch a plane), feedback welcomed.

Last month I had the great pleasure to participate in Agile Testing Days. I did a tutorial on using the Agile Testing Quadrants for test planning, and a keynote on 7 key success factors for agile testing success. More importantly, I got to attend a lot of great sessions by leading agile testing practitioners.

There were so many good sessions and I’ll only mention a few here. Mark your calendars for next year’s Agile testing Days, Oct. 4 – 6.

My slogan after this conference comes from Dave Evans (and from Mike Scott – not sure which one of them coined the phrase but they both talked about it): Respect the Tests. More about that in a minute.

Elisabeth Hendrickson always inspires me, and her keynote didn’t let me down. I’ve heard her definition of agile before, but it bears repeating: Delivering value in the form of releasable software at frequent, regular intervals at a sustainable pace, while adapting ot the changing needs of business. She quoted Mary Poppendieck as saying, if you want to go very fast, you have to be very disciplined. Sustainable pace means that we keep technical debt down so that the code doesn’t impede us.

Elisabeth presented her four sources of technical risk: Ambiguity, assumptions, dependencies and capacity, and her 7 key testing practices: TDD, ATDD, Exploratory Testing, Automated System Tests, Collective Test Ownership, CI, and one more that I think is automated unit tests (I can’t read my notes!)

Elisabeth also quoted Tobias Mayer, who refers to the Product Owner as the Voice of the What, and the dev team as the Tribe of the How. I like this, but I take issue with it a bit. The Poppendiecks have influenced me to experiment with having the dev team learn the business inside out, so the dev team itself can help the business prioritize and even think of features. This worked well for my team at ePlan, and I have to agree that funneling all the business requirements through the Product Owner has a lot of drawbacks.

What was new to me in this talk was something that several of us had talked about a night or two before, the idea of Collective Test Ownership. This ties in with the Whole Team Approach. Everyone on the team takes responsibility for quality, testing, and test automation.

Another great session, again from someone whose work I’m familiar with, was David Evans‘ talk. One big takeaway for me was this: If a test is worth writing, it’s worth automating, and it must always pass in the future. I often see teams that ignore failures in their CI tests because “We have to get these new stories out, this is the PO’s priority.” So the PO doesn’t care whether what she prioritized in previous sprints still works or not? When the CI breaks, the team should stop the line and get it green.

Dave pointed out that if all tests pass, you don’t need extra metrics. Defects are evidence of missing tests. So don’t call it a defect – call it a failing test, write the failing test and fix it right away. If the customer really doesn’t want it fixed now, write a story for later.

Dave recommend fixing time and quality, but don’t fix functional scope. If time is running out, thin the requirements. It’s not like leaving a wheel off or leaving out the engine. Respect the tests – any test worth writing is worth running all the time.

Dave presented his Agile Quality Framework. I won’t go into all the details, but one of my favorite parts is applying balloon patterns. An empty balloon is still a balloon, but if you don’t have the rubber outside, it’s hard to have the air and then add the rubber later. Do the rubber first – the CI, the test framework.

Mike Scott elaborated on the balloon pattern in his demo of Testify Wizard. In one hour, Mike went from zero to a full CI process, including the project two continuous builds, one running unit tests, and one running FitNesse tests. It even generated code coverage reports! I think he did this with both Java and .Net during the same session. He did this with a couple of pushes of the button,some minor configuration and a wave of his crutch.

This shows how easy it is to respect the tests. It’s not that hard to build a framework to integrate our code and run our automated tests so that we get quick feedback all the time.

I’m doing a tutorial and a keynote at Agile Testing Days in Berlin. I think this is the first big agile testing conference to be held in Germany. Take a look at the program – it’s an awesome lineup of agilists. And since the most helpful part of conferences is always the informal connections you make, I’m happy to say that I know of a lot of interesting people planning to participate also.

Here are a couple of discount codes:

10% for the Tutorials: code SPEAKERSTUT010

20% for the conference: Code SPEAKERCON020

I hope to see you there!

If you’re in the Oslo area, don’t forget I am doing an Agile Testing course at Program Utvikling the following week.

There are a lot of creative, smart people on our team! Unfortunately I missed the discussion where these ideas come out, but we are trying some ways to organize such a big team. We will have “cells” or sub-teams, each one will take on a set of stories.

Today, each cell wrote high level acceptance tests together. Our cell started out doing BDD style tests on a FitNesse wiki. We used my desktop and the group in Weston VNC’d in and projected it on screen. But they couldn’t scroll up and down. And tests were taking too long to write. In our pomodoro retro, we decided to do high level test cases together as bullet points only, then have a pair flesh the tests out into BDD style and automate them.

We also decided to try Google docs. We kept using VNC and projecting the screen, but since the page is saved often, each person could also look at the doc on his own machine and scroll up and down. Both changes worked better. I had to split off for a long phone call with someone from an internal development team (that’s a whole ‘nother blog post), and when I came back they had finished the tests.

We are doing what I just learned is set-based engineering to decide whether we’d rather write our SWAT tests in C# and Visual Studio, or in FitNesse. We will do both this sprint and see which is better. My teammate Maykel started writing FitNesse fixtures that would allow using BDD tests to run SWAT from FitNesse, and I think he also had some way to make the tests run faster in FitNesse. One pair will automate the acceptance tests, while the other pair does a spike.

Thanks to the people who commented on yesterday’s post, I feel much better about having the testers get together occasionally. It’s true that we should have a testing community within our agile team, just as we have started one company-wide. I think we are on the right path with this team, despite it being larger than I would like.

I’ve worked on agile teams where programmers, testers, DBAs, sys admins and other roles all saw themselves as part of one developer team for so long, the idea of a separate test team gives me a chill. Yet even on experienced agile teams, there are still testers who see themselves as a “test team”.

I need to investigate more what they mean by this, I might be misinterpreting it. My new team is the combination of three teams, totalling 17 people (not counting the business folks). I think that’s unwieldy. Today, an experienced tester on my new team, who has worked on an agile team for some period of time (I’m not sure how long), proposed that the “test team” get together and discuss practices and strategy. I about blew a gasket, but that didn’t keep the meeting from happening (I’m always on phone and video with my team, so I can’t hep being in the meetings.)

The content of the meeting was reasonable enough. One topic was patterns of tests that we do, for example, for a numeric field you would always try certain test cases such as alpha and special characters, null, etc. Standard stuff.

The next topic was about automating tests on a page and end-to-end level, in addition to at the story level. This bugged me. If you tie tests to a page in the UI, and later stuff gets moved around so that what used to be on that one page is now on different pages, your tests will break. As far as automating end-to-end tests. we definitely have to be careful since the ROI can be bad on automating those.

My teammate Chris McMahon explained how our team up to now has been trying to group tests at the domain level. One FitNesse test contains all tests related to a rating scale, for example, until there are about 200 test steps and then we split it up into some reasonable sub-grouping. We’re testing at a feature level, no matter where we are in the UI.

The next topic was localization. Our app accommodates data in three languages. Someone proposed that we design our tests so that they could be run for any language. This sounds good to me, but how hard is it to do, and is it worth it? Is the localization already tested by the team that does the framework for it? I suggested we definitely needed programmer agreement for this topic, and everyone agreed.

I then brought up topics I’ve been wanting to raise, the non-functional testing. Our company has teams to do performance, scalability, reliability and security testing, but isn’t our team still responsible to make sure those happen in a timely manner? This discussion proved helpful for me, as I learned about the teams that do this testing, and how we can become involved and do a lot of it ourselves. I’m glad to get these activities out in the open and on our minds so they won’t be overlooked.

In the end I needn’t have been so spooked by the idea of testers meeting together. As it turned out, the programmers got into a design discussion with the database guy, and we were not involved in that (welll, I didn’t know about it until it was underway), so it was an opportunity for the testers to talk. But I don’t want that to happen too often.

I’m looking forward to starting work on stories. I think our test design will evolve as we write new code. We will pair on everything, so I hope that means the programmers and testers will remain engaged with each other.

It’s so cool to know that Janet and I have communicated clearly enough what we hoped everyone would take away from our book. Paul Grenyer’s review is a thrill for me.

My teammates Maykel Suarez, J.P. Erkelens and  Eddy Lara ought to be the ones blogging about this as they come up with these cool ideas, but I have the time and the inclination so I will. I hope they’ll read this and correct any errors I make.

I’ve used FitNesse since 2003 and obviously have not learned enough up to now!

We want our FitNesse tests to be DRY (Don’t repeat yourself), easy to maintain and easy to read and understand. Maykel and I were pairing on some FitNesse/SWAT tests this morning and found we needed the same test steps in more than one place. Maykel created this page under our “HelperMacro” area. Each FitNesse variable contains a SWAT test table.

!define NavigateToRatingScale (
!|SWAT |
|StimulateElement|Expression|

I’ve been working in this large agile setting (28 Scrum teams, 2 week sprints, 4 production releases per year) about 6 weeks now. It has confirmed my past experiences that success depends on having good people, and letting them do their best work.

In this short time, my 5-person team (3 programmers, 2 testers) has implemented a CI and build process in Hudson, which build our code as well as the full product, runs our automated unit and FitNesse/SWAT tests, deploys to our sandboxes, and reports results. We’ve finished several stories. We’ve had one sprint review, which went well. We’ve shaved a lot of yaks, which is to be expected for a new team and new employees (especially yours truly. The combination of telecommuting and lack of Windows and network expertise isn’t good). We work in pairs and trios, and stay in constant communication via Skype, chat, webcam and shared desktops.

Something new to me is having to depend on other teams for certain tasks. If we need something added to the database, we have to put a request into Jira and wait, though the turnaround is fairly quick. In order to get data changes into our own environments, we sometimes have to do a db refresh process that can take hours. We even had to wait a few days for another team to provide the new error message we needed. Before we can get final approval from our PO on a story, we have to request that the official company-wide build be deployed on one of our environments. I was frustrated by this, but we seem to be adjusting our rythym so that we fill in the delays with productive work.

To give you an idea what my job is like now, here’s a typical day in my work life:

Before 8:00 AM Mountain time – Get on group chat with rest of team. I’m the last one on – 3 are on Eastern time, and 1 is just faster at getting to work than I am.

8:00 AM – Attend Scrum of Scrums by conference call. This consists of general news that affects all teams, such a build problems, and polling each of 28 teams for impediments. It’s usually over in 10 minutes, amazingly.

8:15 – Join Skype call and webcam videos with rest of team. Pair with fellow tester (Chris McMahon) to write automated GUI/functional tests in FitNesse/Skype. We use VNC to share desktops.

9:30 – Standup meeting. We have an app that brings up a photo of “The Usual Suspects” with our heads on random bodies. I look good with a tattoo.

9:45 -10:00 – Watch the developers work on some unit tests, using LiveMeeting to see their desktop.

10:00 – 11:30 – Catch up on emails, maintenance tasks, and the like, and eat lunch.

11:30 – we’re all back on Skype, chat, webcam and desktop sharing.

11:30 – 12:00 One of the developers helps me with an issue I’m having with my local test environment, using VNC to take control of my desktop.

12:00 – 12:30 We discuss our acceptance tests with the PO. The other teams he works with put acceptance tests in spreadsheets and attach them to the story “card” in Jira. We are writing the acceptance tests together in sprint planning, in BDD style, directly onto the FitNesse wiki pages where we’ll also write the automated tests. Then Chris and I automate the tests. The PO is fine with our approach, he can easily understand the tests, and has found some test cases that we missed. However he’s concerned that we’re deviating from the “standard”. So far, nobody has objected to our writing tests in the FitNesse wiki, and it works so well for us, we’re going to continue.

12:30 – 1:30 Pair with Chris again to update the narrative acceptance tests for the stories based on the PO’s input, and write additional automated tests.

1:30 – 3:00  The database change needed for the story the developers finished yesterday is ready. Chris and I update our local databases, deploy the new code, run the FitNesse/SWAT tests, and do exploratory testing of the new functionality. Whenever we run across an issue, we show it to the developers using VNC desktop sharing.

3:00 – 5:00 Everyone else is done at 3:00, so I’m on my own. I update the wiki with notes about how to use the system that allows us to add test users. Then I do some additional exploratory testing on the story we worked on earlier. Next, I look into a problem we were having in a SWAT test. I had emailed the internal user group about it, and had some responses with suggestions to try. I’m able to get this working, and check the test in. I make sure our task board is up to date.

As time goes by, I spend less time on things like getting my environment working or solving network problems, and more time doing productive work. I’m also starting an internal company testing community, so we can all share ideas. My teammates and I are blogging internally about how we are writing and designing tests, and how we’ve implemented our CI and build process. We hope the ideas that work for us will spread to other teams, and we’re also looking at what other teams to and adopting their good ideas.

So far, so good!

Last week I attended SEETEST in Sofia, Bulgaria. The conference was extremely well-organized, and I learned some new things, such as test automation considerations for testing mobile devices, and coordinating agile testing with compliance requirements. There were also some things said that hit one of my hot buttons: whether testers and programmers ought to work together, versus on separate, siloed teams.

This inspired a rant which I posted (OK, Joey McAllister posted it for me) on my StickyMinds blog entitled Independent Testers? or Independent Thinkers?. I would love to have comments on your experiences with testers and programmers working together.

Coincidentally, the latest Methods and Tools magazine has an article I wrote on Coding and Testing: Programmers and Testers Working Together. In this article I work through an example of how testers and programmers collaborate to develop new features, similar to the examples Janet and I use in our book. Please check it out along with the other interesting articles in Methods and Tools.

I’m working with programmers and testers together more than ever on my new team. Right now we are all on a voice chat, with a shared desktop, writing FitNesse/SWAT tests and code together. (OK, I am really just watching, but I’m learning, and asking questions.)

I’ve worked on four agile teams in the past nine years. Three were highly successful, especially the one I was on for the past 5.5 years. One never really made the transition, though we had a successful agile project. One of the successful ones involved a team of about 30 developers, and that’s the largest project I’ve been on up to now.

I could have happily worked with my previous team until I retire, but you know me, I cannot resist new opportunities. So now I’m on a small Scrum team – that is one of 28 Scrum teams at my new company! Yes, this company actually manages to have 28 teams, including some which provide infrastructure such as frameworks and deployment, working together to release a high-quality commercial software product four times a year.

Yeah, back in the day, XP was for little, co-located teams. “You can’t do shrink-wrapped software with XP” was just one of the many axioms I heard.

Now I work in an agile organization with about 140 developers, some (like me) in remote locations, delivering software both for Windows client and for a hosted solution. (I don’t know everything about it yet so sorry to sound vague!) If you had told me this five or ten years ago, I wouldn’t have been sure it could work.

Since my new 5-person team is a self-organizing Scrum team, we picked our own ScrumMaster, which ended up being me. I love being an uncertified ScrumMaster, and plan to delegate a lot, because I’m also committed to remaining a hands-on tester. I attended my first Scrum of Scrums (some attendees called it the “SoS” which I loved) on Friday. Each of the 28 ScrumMasters (some in the room, some on the phone) was called on to say if they had any impediments. This happened to be a release day, so I thought there might be lots of impediments, but most teams didn’t have any. Those that did got help right away. The meeting was over in 10 or 15 minutes.

Granted, not every team is high-performing, and some practices I’m used to are missing altogether. But think about it, a software development organization that large, which only implemented Scrum four years ago, able to coordinate and release critical value frequently, at a sustainable pace. That’s agile in anyone’s book.

Follow this space for my adventures as I enter a whole new world!