Agile Testing with Lisa Crispin

Eric Willeke, Liz Keogh and Jean Tabaka have inspired me (and many others, I am sure) with the following, which they produced when they got together in Boulder last Friday.

I’m currently reading some draft chapters of Jurgen Appelo’s upcoming book. His work is affirming my belief that successful software projects are the result of getting good people and letting them do their best work. There are so many aspects to this – reading Jurgen’s work, I am starting to understand that people are the most complex component of any project! Everything hinges on our ability to innovate, to be creative. So, we must be a community of thinkers.

One item in this list urges us to embrace newcomers. I was welcomed into the agile community 9+ years ago. I work hard to pay that nurturing forward, so that others may experience gaining knowledge that in turn drives their creativity. And besides all that – why not always be nice to each other? I feel another blog post coming on.

A Community of Thinkers

I am a member of a community of thinkers.

I believe that communities exist as homes for professionals to learn, teach, and reflect on their work.

I challenge each community in the software industry to:

• reflect and honor the practitioners who make its existence possible;
• provide an excellent experience for its members;
• support the excellent experience its members provide for their clients and colleagues in all aspects of their professional interactions;
• exemplify, as a body, the professional and humane behavior of its members;
• engage and collaborate within and across communities through respectful exploration of diverse and divergent insights;
• embrace newcomers to the community openly and to celebrate ongoing journeys; and
• thrive on the sustained health of the community and its members through continual reflection and improvement.

I believe that leaders in each community have a responsibility to exhibit these behaviors, and that people who exhibit these behaviors will become leaders.

I am a member of a community of thinkers. If I should happen to be a catalyst more than others, I consider that a tribute to those who have inspired me.

”A Community of Thinkers” by Liz Keogh, Jean Tabaka and Eric Willeke is licensed under a Creative Commons Attribution-Share Alike 3.0 License. Please attribute to the distributor of your copy or derivative.

Beautiful Testing, with a wide variety of fascinating chapters contributed by leading software professionals, is available. Not only will it inspire you and your team to improve your testing and your product, but all royalties go to a good cause. I’m honored to have written a chapter. Please see the page on my site for more details and a free copy of my chapter – to whet your appetite for the rest of the book!

I love conferences – I get to meet so many smart practitioners and come home bursting with new ideas. I’m back from Agile Development Practices and I learned a ton. One thing that’s on my mind both from the conference and other reading I’ve done this week is the importance of a culture that enables and even promotes learning.

It’s easy to forget that failure is good – if you can fail fast enough. That’s one reason agile works so well – short iterations and work transparency let us fail fast, and everyone knows when we fail. But failure shouldn’t only be seen as negative. We learn a lot more when we fail.

I recently read a great post titled “Are you building a learning suppression system by Daryl Kulak from his upcoming book. Here’s an excerpt.

What happens in organizations is that people get punished for committing sins of commission but they do not get punished for sins of omission. This shapes the mind of a person into saying “No, we better not try that,” attempting to avoid the nasty results of commission errors while ignoring the problems of omission errors, which don’t seem to cause as much havoc.

We don’t want to suppress innovation. Mistakes should be tolerated, not punished. We want a learning organization so our teams can always adapt to change, and thrive.

More on Learning from Linda Rising

Linda gave an enlightening talk Wednesday at ADP about retrospectives, and she talked a lot about having time for thinking and learning. These are my notes on her talk, so I hope I haven’t misinterpreted anything she said. You can find more material direct from her on her site.

When confronted with practices such as pair programming or retrospectives, some managers have this reaction: “We don’t have time for thinking”.

But we aren’t trying to stop, analyze the root cause of all of our problems, and fix them all. We need to do little experiments to see if they solve some problem we’re having. If the solution works, we don’t even have to know why, we can just use it. Linda compared this to tuning a musical instrument – we can make tiny changes, and these give us lots of value.

Linda quoted Tom DeMarco’s book, Slack. You can’t learn or change if you have no slack. Linda cited research showing that not being able to express feelings and thoughts in the normal course of business weighs us down, slows us down. It sounded a lot like technical debt to me! We fret about things that aren’t going well, and if we don’t get a chance to experiment with ways to improve them, we become less and less productive, more and more unhappy.

Going back to that idea that we must tolerate failure, it’s important that we don’t criticize or blame our teammates. Every team member should feel safe to raise issues and propose ideas. Linda cited Norm Kerth’s prime directive for retrospectives: We must believe that everyone did the best job he or she could, given what was known at the time. Everyone used their skills and abilities, the available resources, to their best advantage, given the situation at hand. When we start from there, we can avoid pointing fingers, and simply try to learn: what’s going well? What’s not going so well? How can we improve? What still puzzles us?

Linda advises us to watch for stereotypes: “These people are just no good at…” She was talking in the context of retrospectives, but this is good advice for everyday work. We even stereotype ourselves. We have to believe, instead, that we can get better. Then we will.

Focus on experiments. Identify the next experiment, and ask the three questions about it at the next retrospective. Share knowledge. Look for patterns.

Time to Learn

As Linda noted, everyone says they want to learn, but few take the time to do so. Take some time to learn something new today. See what you can do to promote a learning culture within your organization. Show your manager the Google 20% rule, and the value that Google has derived from that! Lots of companies, big and small, have something similar: Wiki Wednesdays, Engineering Sprints, one hour per day devoted to professional growth. People on agile teams love learning, and it makes us happy. When we’re happy, and when we are always learning and experimenting, we can do our best work. To me, that’s what agile is all about.

Last month I had the great pleasure to participate in Agile Testing Days. I did a tutorial on using the Agile Testing Quadrants for test planning, and a keynote on 7 key success factors for agile testing success. More importantly, I got to attend a lot of great sessions by leading agile testing practitioners.

There were so many good sessions and I’ll only mention a few here. Mark your calendars for next year’s Agile testing Days, Oct. 4 – 6.

My slogan after this conference comes from Dave Evans (and from Mike Scott – not sure which one of them coined the phrase but they both talked about it): Respect the Tests. More about that in a minute.

Elisabeth Hendrickson always inspires me, and her keynote didn’t let me down. I’ve heard her definition of agile before, but it bears repeating: Delivering value in the form of releasable software at frequent, regular intervals at a sustainable pace, while adapting ot the changing needs of business. She quoted Mary Poppendieck as saying, if you want to go very fast, you have to be very disciplined. Sustainable pace means that we keep technical debt down so that the code doesn’t impede us.

Elisabeth presented her four sources of technical risk: Ambiguity, assumptions, dependencies and capacity, and her 7 key testing practices: TDD, ATDD, Exploratory Testing, Automated System Tests, Collective Test Ownership, CI, and one more that I think is automated unit tests (I can’t read my notes!)

Elisabeth also quoted Tobias Mayer, who refers to the Product Owner as the Voice of the What, and the dev team as the Tribe of the How. I like this, but I take issue with it a bit. The Poppendiecks have influenced me to experiment with having the dev team learn the business inside out, so the dev team itself can help the business prioritize and even think of features. This worked well for my team at ePlan, and I have to agree that funneling all the business requirements through the Product Owner has a lot of drawbacks.

What was new to me in this talk was something that several of us had talked about a night or two before, the idea of Collective Test Ownership. This ties in with the Whole Team Approach. Everyone on the team takes responsibility for quality, testing, and test automation.

Another great session, again from someone whose work I’m familiar with, was David Evans‘ talk. One big takeaway for me was this: If a test is worth writing, it’s worth automating, and it must always pass in the future. I often see teams that ignore failures in their CI tests because “We have to get these new stories out, this is the PO’s priority.” So the PO doesn’t care whether what she prioritized in previous sprints still works or not? When the CI breaks, the team should stop the line and get it green.

Dave pointed out that if all tests pass, you don’t need extra metrics. Defects are evidence of missing tests. So don’t call it a defect – call it a failing test, write the failing test and fix it right away. If the customer really doesn’t want it fixed now, write a story for later.

Dave recommend fixing time and quality, but don’t fix functional scope. If time is running out, thin the requirements. It’s not like leaving a wheel off or leaving out the engine. Respect the tests – any test worth writing is worth running all the time.

Dave presented his Agile Quality Framework. I won’t go into all the details, but one of my favorite parts is applying balloon patterns. An empty balloon is still a balloon, but if you don’t have the rubber outside, it’s hard to have the air and then add the rubber later. Do the rubber first – the CI, the test framework.

Mike Scott elaborated on the balloon pattern in his demo of Testify Wizard. In one hour, Mike went from zero to a full CI process, including the project two continuous builds, one running unit tests, and one running FitNesse tests. It even generated code coverage reports! I think he did this with both Java and .Net during the same session. He did this with a couple of pushes of the button,some minor configuration and a wave of his crutch.

This shows how easy it is to respect the tests. It’s not that hard to build a framework to integrate our code and run our automated tests so that we get quick feedback all the time.

There’s so much swirling around in my brain from a whole week in Chicago at Agile 2009. Let me start with my recollections of the Agile Alliance Functional Test Tools workshop. Jennitta Andrea chairs this effort, and Elisabeth Hendrickson facilitated this meeting.

The facility was great, plenty of room and good connectivity, and had a nice view of the Sears (Willis) Tower and a sliver of lake!

The workshop was all Open Space, so we started out by planning the sessions. 3 timeslots, 4 locations, though there ended up being only 11 sessions. One of the first sessions was lightning talk demos of various tools – Canoo WebTest, Twist, Cucumber, Robot Framework and SWAT. (Not to say that Watir and Selenium were ignored – we were just already familiar with those). I was impressed by Robot Framework, an open source tool whose developer is Pekka Klark. It has a lot of flexibility. It uses a tabular format similar to FitNesse except there is only one type of table. You can do keyword driven, data driven and even BDD style tests. It also takes command line arguments. It lets you have internal and external libraries such as Swing.

Mike Longin (who works for the same company I do) did a great job demo-ing both SWAT and UltiFit (the latter is not open source, but as there was interest in it, Mike may undertake that task). There was a lot of interest in SWAT, nobody had seen it before and I think they were particularly impressed by the IDE.

One of the other morning sessions was how to get Cucumber to work with .Net. Of course you can run Cucumber tests on the browser no matter what your app language, but if you want to test in the same process as your app code, you need some bridge. After a lot of discussion of things like Iron Ruby, someone had the idea to use the Slim protocol between Cucumber and .Net. Aslak Hellesoy, the Cucumber developer, thinks this idea may have legs. I have some photos of the diagrams they drew and will upload them here when I get time.

Paul King, one of the WebTest contributors as well as the main Groovy developer, proposed doing more mixing and matching of testing frameworks/drivers/utilities/tools. He had a nice graphic showing what is available. We all agreed there are enough test runners, and the developer community should focus on solving new problems. Gerard Meszaros created a spreadsheet showing all the tools along with the features and requirements people might have for their test framework/drivers. Some of us got together in an afternoon session to tweak the tools spreadsheet and Gerard posted it on Google docs so that tool developers/users can add to it and fill it out. This is the start of a great resource both for users looking for the right tool to developers who want to know what they might be able to build on or adapt.

My teammates Maykel Suarez, J.P. Erkelens and  Eddy Lara ought to be the ones blogging about this as they come up with these cool ideas, but I have the time and the inclination so I will. I hope they’ll read this and correct any errors I make.

I’ve used FitNesse since 2003 and obviously have not learned enough up to now!

We want our FitNesse tests to be DRY (Don’t repeat yourself), easy to maintain and easy to read and understand. Maykel and I were pairing on some FitNesse/SWAT tests this morning and found we needed the same test steps in more than one place. Maykel created this page under our “HelperMacro” area. Each FitNesse variable contains a SWAT test table.

!define NavigateToRatingScale (
!|SWAT |
|StimulateElement|Expression|

I’ve been working in this large agile setting (28 Scrum teams, 2 week sprints, 4 production releases per year) about 6 weeks now. It has confirmed my past experiences that success depends on having good people, and letting them do their best work.

In this short time, my 5-person team (3 programmers, 2 testers) has implemented a CI and build process in Hudson, which build our code as well as the full product, runs our automated unit and FitNesse/SWAT tests, deploys to our sandboxes, and reports results. We’ve finished several stories. We’ve had one sprint review, which went well. We’ve shaved a lot of yaks, which is to be expected for a new team and new employees (especially yours truly. The combination of telecommuting and lack of Windows and network expertise isn’t good). We work in pairs and trios, and stay in constant communication via Skype, chat, webcam and shared desktops.

Something new to me is having to depend on other teams for certain tasks. If we need something added to the database, we have to put a request into Jira and wait, though the turnaround is fairly quick. In order to get data changes into our own environments, we sometimes have to do a db refresh process that can take hours. We even had to wait a few days for another team to provide the new error message we needed. Before we can get final approval from our PO on a story, we have to request that the official company-wide build be deployed on one of our environments. I was frustrated by this, but we seem to be adjusting our rythym so that we fill in the delays with productive work.

To give you an idea what my job is like now, here’s a typical day in my work life:

Before 8:00 AM Mountain time – Get on group chat with rest of team. I’m the last one on – 3 are on Eastern time, and 1 is just faster at getting to work than I am.

8:00 AM – Attend Scrum of Scrums by conference call. This consists of general news that affects all teams, such a build problems, and polling each of 28 teams for impediments. It’s usually over in 10 minutes, amazingly.

8:15 – Join Skype call and webcam videos with rest of team. Pair with fellow tester (Chris McMahon) to write automated GUI/functional tests in FitNesse/Skype. We use VNC to share desktops.

9:30 – Standup meeting. We have an app that brings up a photo of “The Usual Suspects” with our heads on random bodies. I look good with a tattoo.

9:45 -10:00 – Watch the developers work on some unit tests, using LiveMeeting to see their desktop.

10:00 – 11:30 – Catch up on emails, maintenance tasks, and the like, and eat lunch.

11:30 – we’re all back on Skype, chat, webcam and desktop sharing.

11:30 – 12:00 One of the developers helps me with an issue I’m having with my local test environment, using VNC to take control of my desktop.

12:00 – 12:30 We discuss our acceptance tests with the PO. The other teams he works with put acceptance tests in spreadsheets and attach them to the story “card” in Jira. We are writing the acceptance tests together in sprint planning, in BDD style, directly onto the FitNesse wiki pages where we’ll also write the automated tests. Then Chris and I automate the tests. The PO is fine with our approach, he can easily understand the tests, and has found some test cases that we missed. However he’s concerned that we’re deviating from the “standard”. So far, nobody has objected to our writing tests in the FitNesse wiki, and it works so well for us, we’re going to continue.

12:30 – 1:30 Pair with Chris again to update the narrative acceptance tests for the stories based on the PO’s input, and write additional automated tests.

1:30 – 3:00  The database change needed for the story the developers finished yesterday is ready. Chris and I update our local databases, deploy the new code, run the FitNesse/SWAT tests, and do exploratory testing of the new functionality. Whenever we run across an issue, we show it to the developers using VNC desktop sharing.

3:00 – 5:00 Everyone else is done at 3:00, so I’m on my own. I update the wiki with notes about how to use the system that allows us to add test users. Then I do some additional exploratory testing on the story we worked on earlier. Next, I look into a problem we were having in a SWAT test. I had emailed the internal user group about it, and had some responses with suggestions to try. I’m able to get this working, and check the test in. I make sure our task board is up to date.

As time goes by, I spend less time on things like getting my environment working or solving network problems, and more time doing productive work. I’m also starting an internal company testing community, so we can all share ideas. My teammates and I are blogging internally about how we are writing and designing tests, and how we’ve implemented our CI and build process. We hope the ideas that work for us will spread to other teams, and we’re also looking at what other teams to and adopting their good ideas.

So far, so good!

If you’re wondering whether our book is for you, read this comprehensive review by Brad Appleton in the Agile Journal. Janet and I are so excited that so many readers understand what we have tried to do with the book. We wanted to give pragmatic, practical help for agile testers and teams, and that’s what people say they are finding. Yay!

If you have read our book, please review on Amazon or your blog, and help other potential readers decide if it’s the right book to help with their issues.

My team was lucky in having Brian Marick visit us last week, interviewing some of us for his book “Travels in Software”. He had visited us back in 2004, and I hope it was interesting for him to see that 1) we used a lot of ideas we got from him, such as using examples to drive coding and 2) we’ve changed and grown a lot and are a productive team.

It was fun to be interviewed by Brian, and he asked thought-provoking questions. One question was, what would you go back and tell your 2003 self, when you first started on this team? Or presumably, even my 2000 self on my first agile team. This has started a couple different thoughts in my head.

Here’s one thing I’d be sure to tell myself. “Lisa, you are good at picking up a new domain, and you’ll learn about 401(k) plans, all their government regulations, how all the different users use the application and other knowledge directly related to the application under test pretty fast. However, there’s lots more to know than you think. You’ll still be learning about parts of the app you never saw before in 2009! Also, you’d better learn about the whole business, not just the parts that are automated in the app.”

Since Day 1 of implementing Scrum, our team has been committed to delivering the highest possible quality software, and I think we’ve done a fairly good job. However, one of our biggest problems has been that we sometimes misunderstand customer requirements, or miss requirements entirely, even though we are driving development with tests.

It was only after we made a conscious, concerted effort to sit with the business people, learn all their processes in detail, and document them, that we got a full enough understanding of the business. For example, only recently did I learn about all the different bank accounts through which money moves around, because much of the movement is done manually. It’s critical to know this, however, because it’s very bad when cash balances stop reconciling correctly, or when the clearinghouse or bank says we have a different amount in each account than we think we do.

Maybe it was important that we first master agile practices such as TDD, CI, refactoring, driving coding with examples and tests at all levels, and test automation, before we moved on to getting a broad and deep understanding of the business and of every person’s job. You can’t do everything at once. Still, it would have helped me to at least have a sense of what I didn’t know yet, so I didn’t have a false sense of confidence.

Do you know your business inside and out? Or only the application for which your team writes code?

I am extremely proud that our book is #28 on Jurgen Koop’s list of the top 50 new software development books! Also flattered to be in such excellent company on this list. There are some titles there that I hadn’t heard about before – time to go order some books! Check out the list and see what’s missing from your pile of books to read.