Agile Testing with Lisa Crispin

My friend and excellent agile coach Michele Sliger, co-author of The Software Project Manager’s Bridge to Agility ,  recently emailed me this question that she hears a lot: “Where should I start looking to learn more about test automation and tooling?”

Naturally, I replied with a typically long-winded answer. She must have liked it though, because she suggested I cut and paste my reply into a blog post. Here it is!

The key thing with a newbie agile team is to not say “move testers on all teams into automated testing” but “help all teams learn to take whole-team responsibility for quality and testing, and learn how programmers and testers collaborate to automate tests at all levels”. Automated test code is code, and generally, it’s much quicker and better in the long run for programmers to code the automation fixtures. Testers know what to test, and when programmers do the coding, testers have lots of time to help customers come up with examples of desired behavior to turn into tests. Then they can pair with the programmers to automate regression testing and any other types of tests (performance, load etc). Then they will still have time to do the all-important manual exploratory testing.

If they read our book Agile Testing, they will learn that they have to be patient and invest a lot of time in learning and experimenting with different test frameworks. And they should use Mike Cohn’s test automation pyramid concept to see where they’ll get the best ROI in automation. Generally, they should always start by learning how to do TDD and get traction on unit level tests. Then they can move on to API or service level tests, then GUI. But that’s not a rule, each team has to figure out what’s best for them.

I highly recommend Gojko Adzic’s Specification by Example book, and also his blog, gojko.net. Markus Gaertner has a great ATDD book coming out but it’s not published yet.

There are SO many test frameworks, drivers, tools, it’s overwhelming. Recently we had to find a way to automate GUI regression tests for our new code that uses Dojo. Our existing GUI tools aren’t able to interpret the Dojo JS properly. It was truly a team effort, though not a smooth one. Our sys admin had earlier played around with Selenium, and we thought Webdriver might work. The sys admin did a spike and proved it did work, then he and our senior architect spiked a couple of different frameworks – one homegrown, one using Geb. They demo’ed it and got everyone’s opinions. For now we are going with the Geb framework, but down the road we might decide it’s not quite right and try something else.

Anyway, that’s a long-winded answer. One good place to start getting an idea about functional test tools is the Agile Alliance Functional Test Tools group’s spreadsheet comparing various tools, at http://bit.ly/AgileTestTools. When the teams are ready to think about functional testing, they might take a look and get an idea of all the choices. It’s a good idea to stop and plan a test automation strategy.

It is really, really hard to get started with test automation. There’s a period where it’s just extra work and no reward. But eventually they’ll cross over that Hump of Pain and start reaping the benefits, eating away at their technical debt!

I was pleased to learn that my StarEast talk on an agile approach to defect management generated more discussion on the topic at the conference. Please read Gojko Adzic’s new post “Bug statistics are a waste of time” – it shatters a lot of illusions that organizations treasure about defect tracking and bug metrics.

I’ve just been struggling with a title for this post. Some of my ideas were: “Visibility Taken to New Heights”, “Yes, There Are Teams Who Do This Just Like In the Books”, “A Real Commitment to Continual Improvement”, “Inspiration”, but none of them really capture my amazement at what I saw visiting the team at Energized Work in London. Finally it struck me that this team embodies the Whole Team approach to software development in a way that I’ve rarely seen.

I’ve just updated this post with links to photos of the Big Visible Charts in the Energized Work Lab, accompanied by explanations from Simon Baker. Check ‘em out, they will give you ideas for your own team to use.

What a Team!

I’ve always maintained that I work for the coolest team on the planet, and most of what I try to help others learn are things I learned from and with my own awesome teammates. I don’t want to be disloyal, but the team at Energized Work raises the bar for cool. Simon Baker (@energizr on Twitter) invited me to visit while I was in London earlier this month (and he gave me permission to write this post about it). I love to visit development teams wherever I go, and learn what other practitioners do to improve software quality, so I was happy I could make the time. Mohinder Kohsla had breakfast with me that morning, and was kind enough to guide me to the Energized Work lab, as he already knew some of the folks there.

Big Visible Charts

What blew me away right off the bat at Energized Work was the creative ways they use whiteboards. One whiteboard had personas, complete with photos and bios, and assumptions. That’s not too unusual, but it’s still nice to see someone actively using personas in real life.

What intrigued me most was the way they represent their backlog for their current project as a site map on a big whiteboard. They start drawing site map for the web application they’re working on on a big whiteboard, with placeholders for each screen, and arrows showing how the navigation flows. Since this is on the whiteboard, it’s easy for them to evolve the map as the project proceeds. They overlay cards with user goals so they can identify which journeys deliver value. Screen prints of completed pages are stuck to the board over the original placeholder, along with the cost to develop it.

The team starts with minimum functionality delivered to achieve a goal, working in fast, tiny iterations. As Simon explained to me, the customer can decide based on that functionality and what it cost whether to invest more in that user goal, for example, to “enrich the feature with more functionality”. The site map / backlog is part of the team’s mechanism which allows the customer to decide “just in time” whether to “go deep or broad” every week in response to user testing.

The Energized Work team has experimented with different ways to do story/task boards as well. Instead of a traditional Scrum task board or Kanban board, they represent each story with a big square on the whiteboard. In the middle of the square is a story card has some high level acceptance tests written on the back. Surrounding it is plenty of space to draw mock-ups, prototypes, write questions, write down test cases, whatever they need to discuss or do. A team member only has to look at the board to see story requirements.

Instead of moving cards from column to column, eg. from “Work in Progress” to “Verify”, they represent the vertical slicing of the story with different colored dots, representing feedback from different activities including customer review, UX reviews, and manual exploratory testing. Some cards may cycle through coding, testing, reviews, and back to coding several times before they’re done. Gordon Conroy, the tester, keeps an eye on the “big picture” as new functionality is created slice by slice. He was so knowledgeable about all aspects of the project, it’s clear he works constantly together with the rest of the development team. If companies with separate test teams could see this in action, they’d get why testing can’t be a separate phase or done by a separate team!

My own team uses a fairly standard task board with rows for stories and columns for status, and we write high-level requirements and questions on a separate whiteboard. We also work in these fast, tiny iterations with multiple slices of each story, and I am wondering if we can borrow some of these ideas to better represent that visually. We put more details about slices, test cases and specifications on the team wiki, but developers don’t always look at the wiki. Of course we also do specification by example with executable tests, but whiteboard drawings and notes would make a good, quick-to-access supplement during development. I’ve told my team everything I’m writing here, and we’re ruminating on how we might experiment to improve visibility, while keeping our remote teammate involved.

Another Big Visible Chart in the Energized Work lab lists usability heuristics. Clearly, no aspect of software quality is neglected here, and there were so many visual reminders to keep the team on track. Just about every problem my team has experienced, we solved by making it more visible, and seeing how well another team accomplishes this is affirming.

Driving Development with Tests

The Energized Work team are clearly expert practitioners of TDD and specification by example. They showed me the continuous builds in Jenkins for a couple of different projects, it looked a lot like what my own team does. I’ve personally met few teams that have as sophisticated a build job set-up as ours. I found it interesting that they’ve used different test frameworks and tools on different projects, they are clearly committed to finding what works best for each situation, and experimenting with new approaches. Some of the tools they use are new to me, such as Spock for unit testing Groovy and Java, using a BDD-style syntax.

All database changes and data migrations are automated, kept under source code control and managed with Liquibase. This is one area where my own team could improve, so it was helpful to see this in action.

Gordon Conroy showed me some clever solutions they’ve come up with to test having many concurrent users testing realistic situations, and how they can monitor these scenarios as the automated tests run.

Customer Collaboration

They explained how they work with their current customer. He comes in several days a week to collaborate with all team members, including developers and testers, and answer questions. The customer’s goal is to have a website to show potential investors, so a lot of effort goes into the look and feel and showcasing the functionality. The customer gets continual feedback from tests and from the backlog site map, and in turn is able to review the work completed so far and give feedback to the development team on what changes are still needed. I wish I could have met the customer because I’m betting he is truly delighted (as we want all of our customers to be!)

Continual Improvement

Most of all, I admire the Energized Work team’s obvious commitment to always finding better ways to work, even though they are already functioning at such a high level. They had just experimented with using causal loop diagrams for their retrospectives, to help them identify behaviors and invisible work. They told me that they aren’t just trying to improve the process within the system, but better understand the system itself. As a result, they’re able to make real improvements in effectiveness, not only efficiency. Here’s another example of output from a retrospective.

It was exciting to meet a team of people who are passionate about software quality, love what they do, and clearly have a lot of fun. They obviously have time to learn, innovate and experiment, which is reflected in some incredibly creative solutions to some tricky testing problems.

I’ve been telling my own teammates the ideas I brought away from my visit, and we’ll see what experiments we can think of trying. Some of our best ideas were “stolen” from other teams, and I expect in another six months I’ll be able to tell you what has resulted from my inspiring visit to Energized Work.

Everyone Focused on Quality

Janet Gregory and I, and our teams, have been practicing the Whole Team approach to delivering high-quality software for years, and working hard to explain this concept to others. It was affirming to see a team commit to a high standard of quality and then work relentlessly to keep raising the bar.

I highly recommend that you and your team visit other teams, in your own area or when you are traveling. Every team can teach you something, even if it’s “what not to do”. Visiting a team who has found good ways to produce great software shows you what’s possible in real life, and leave you enthused about trying something new to do better work.

Yesterday I had a call from a participant in my recent BayAPLN session. I completely screwed up and, when I tried to save the phone message, I deleted it, and somehow the number that my phone saved, I could not call back. So, I’d like to try to answer the question, as best I remember it, here. If you still have questions, please email me, because this is such an important topic.

Since I listened to this question in my car and could not write it down, I don’t have it exactly, but it was along the lines of: “My team has no automated testing. One of our user stories did not have all the testing completed by the end of the sprint, because there was not time to complete the manual tests. Would you please recommend tools to help us?”

I know of a lot of great testing tools, GUI drivers and frameworks. If you want to do GUI testing, you can’t go wrong with Canoo WebTest, Selenium, Watir. Robot Framework, FitNesse and other frameworks can be used for testing at the API level or combined with a driver for testing through the GUI. Those are just a few examples of what’s available. Even better, your team can write its own testing frameworks and harnesses.

But tools aren’t the point. The point is that the best people to solve your testing problems is your own development team. By “development team”, I mean programmers, testers, BAs, DBAs, sys admins, everyone involved in delivering software. If you aren’t delivering user stories in a sprint because testing isn’t finished, you need to look at that problem as a team. By bringing all your team’s different roles, experiences and viewpoints to bear on this problem, you will think of experiments to try, and you will arrive at solutions.

Give yourselves time to experiment. Engage everyone involved in delivering software in solving the issues of getting all testing activities done each sprint or iteration. This diversity of viewpoints means you will eventually succeed. YOUR TEAM can solve this problem. Give it a chance!

This past Saturday I was fortunate to be one of the 60 people enrolled for the largest Code Retreat ever, Code Retreat Boulder, facilitated by Corey Haines and Chad Fowler, organized by Prakash Murthy. This was my first Code Retreat, though I’ve worked on teams practicing TDD, pair programming, and good design practices for more than ten years.

I was the only tester attending, and as I had feared, I was somewhat out of my depth. Nevertheless, I write automated tests, and I need to apply the same design basics to those as we do to production code. The past couple of years, I’ve tried to think of ways to help testers learn those principles, and written articles such as “An Intro to Test Design” in the Testing Planet. One goal of Code Retreat is stretching yourself, making ugly messes and not worrying about it even though our goal is perfection. I was definitely stretched way beyond my comfort zone!

Corey explained the basics we should practice:

1. Tests pass – J.B. Rainsberger says this is “like breathing”.
2. Don’t Repeat Yourself, keep code DRY
3. Tests and code reveal intent – name things well
4. Small – work in tiny increments.

The day was divided into 45-minute pairing sessions; we switched pairs each time. We wore name tags that indicated our programming language preferences. At the end of each session, we had to delete our code, stand up, and debrief for a few minutes about what we learned and what to try next. We also had a long lunch break (with delicious food!) The coding problem to solve was Conway’s Game of Life. As I dislike both math and puzzles, this scared me.

Mike Clark generously offered to pair with me in the first session. I’ve known Mike for ten years and he’s a TDD guru. We used RSpec and TextMate, both new to me (at work, we use test::unit with Ruby, and I use IntelliJ Idea). It was so natural for Mike to start writing a tiny test, then a tiny bit of code, which he put in the same file to start off – quite efficient. He had brought graph paper, which was handy for writing down examples of what a Game of Life grid would look like. We immediately set about creating and testing a board of cells. Corey came around to look at what we’d done, and commented that we should try to avoid numbers and X/Y coordinates. Hmmm. He mentioned encapsulation, which I understood to mean we don’t need to know about cells, only the board.

Apparently most everyone had started out creating a grid and using X/Y coordinates. For the second session, I paired with Erika Bricey, who works at Rally Software. We decided to follow Corey’s advice to start with the rules, to focus on the names for the methods, and whether the methods are doing too much. Corey mentioned that a class with plural name is a collection of other things. Erika and I had some good ideas on how to approach the problem, but for sure I couldn’t turn those into code, and Erika struggled too. Erika also used RSpec, but she worked in Eclipse.

Seeing our frustration, Corey and Chad suggested we less skilled persons find highly skilled pairs. So I waylaid Marty Haught, a test-obsessed programmer I’ve been wanting to talk to. He went along with my desire to focus on rules and on the names of things. For example, we started out with a module called “alive”, for cells that are alive. Later we needed to have dead cells, so we renamed that to “remain_alive” and the other module was called “dead_cell”. We wrote tests and code for most of the rules. We tried to follow Corey’s advice that “anytime you need to DO something, ASK another object”. For example, to find out if a cell is alive, ask “How many neighbors do I have?” and that will invoke an object that says “Here’s my number of neighbors, am I alive?”

Corey told us that if every time you have to make a decision, you ask someone(thing) else, you end up with no “if” statements. He challenged us to see how far we could go before we had to have a number. He noted that there are two parts to the rule: the hard-coded rule such as “Any live cell with fewer than two live neighbours dies” also states the spirit of the rule: “as if caused by under-population”. Aha! We should write our code in terms of under- and over-population instead of hard-coded boundaries.

My first pair after lunch, Anthony, used MacVim. I’m an old fan of Vim, but hadn’t seen MacVim. We talked about the fact that the grid is infinite, and decided to see if we could get information about the neighbors of any one cell, working with relative numbers. Anthony was pretty sharp at figuring out how to do this, and we made a lot of progress in the 45 minutes. Other pairs tried starting with the outside abstraction and working in.

I was getting tired and didn’t take as many notes in the afternoon – it was all I could do to keep up with what my pair was doing. I had low expectations with regard to my Ruby coding knowledge, but I learned how much I don’t know! For the last session, I was the kid who didn’t get picked for the softball team, I didn’t have a pair, so I observed Rob Park and his pair working in Java. I work on a Java team. It was interesting to see the immediate contrast of working in Ruby vs. Java for the same coding problems. Java looked much slower and more cumbersome to me.

We finished with a closing circle where each person stated what we learned, what surprised us, and what we’ll do differently back at work on Monday. Many of the participants were new to pairing and TDD. I’m not, but it reinforced the value I’ve gotten by learning through pairing with a programmer. I learn more about code design in an hour of pairing than from days of reading books. I plan to be more diligent in applying good design principles to automated test code – for example, pay more attention to using good names. I won’t just take the first solution that occurs to me, I’ll experiment with different approaches. Let’s see if I achieve that goal!

Agile is pretty mainstream now, but some folks still aren’t sure what testers do. In addition, the “Whole Team” approach to quality and the idea that testing and coding aren’t separate phases still mystifies some people. I think my team’s activities the past two days are a good illustration of how agile teams work together to produce high quality software, yet we testers still make valued contributions.

Our software manages all aspects of 401(k) retirement plans. A “plan sponsor” is the company or employer that establishes the 401(k) plan for their employees. We’re working on a theme to allow more than one person working at a company log in and do “plan sponsor” functions for their 401(k) plans, such as submitting payroll contributions or enrolling employees.

Up to now, since only one “plan sponsor” account existed per 401(k) plan, we called that user role “plan sponsor”. This term appears all over our website. Now, we are working on a theme to allow multiple accounts to do the “plan sponsor” functions. These users are not plan sponsors – the employer is. Earlier this week, we had a couple of different discussions on what to call these users and what terminology to use on the user interface. Every manager and developer had a different idea, we kept going round and round discussing it in twos and threes.

Yesterday, I scheduled a meeting with ALL the stakeholders to make a final decision on terminology. You might say, “It’s not the tester’s job to call a meeting like that! Shouldn’t the ScrumMaster or Product Owner do that?” Well, they could, but our ScrumMaster was out sick, and our PO is always swamped. We needed to stop wasting time changing verbiage over and over. In 15 minutes, the various business managers agreed on the term “plan administrator” and on some other related terminology.

Changing terminology - from a mockup

Since the PO was too busy to do it, I mocked up all the changes for all the various pages, posted them on the wiki for our remote developer to see along with narrative requirements and test cases for the changes, and stuck them to our task board along with a task card. Our remote developer, who lives many time zones ahead of us, made most of the changes before we got to work the next morning, and another developer made the rest. I verified the changes, showed the PO, and updated an automated regression test.

We had an issue, though. The modules and classes in our code still used the “plan sponsor” term. We have a “plan_sponsor” table in the database and a column called “sponsor_id”. Eventually it would become very confusing that our UI says “plan administrator” all over the place, and the code has “plan sponsor”. One of our developers decided to copy the “plan_sponsor_mgmt” module to a new module called “plan_admin_mgmt”, and renamed anything in the new module with “plan sponsor” references to be “plan admin” references. This required renaming classes, config files, variables and more. He deleted the code from the old module and will delete the old module itself when the dust settles. (He found out later he could have done the renaming in the old module, but as he says, live and learn).

Later, we will change URLs, velocity variables and spring field declarations, but as that will change HTML Ids and names and cause automated GUI regression scripts to fail, we are coordinating that. And, we still need to change the database table and column names.

It’s a lot of trouble, but it would have been more trouble later, when someone wants to change the “plan administrator” functionality, and can only find “plan sponsor” in the code. We all think we’ll remember everything forever, but we have a big code base considering our team size, and what if we hire a new developer?

So what’s my point? As a team, we did what needed to be done to maintain a high standard of quality, and avoid technical debt. No manager told us to do that. Our Product Owner did not “prioritize” refactoring the code to have the new terminology, because that isn’t a business decision – we control our internal code quality. Though everyone on our team cares about quality, I made my own particular contribution by getting everyone to agree on terminology and verbiage, writing a task card and posting the mockups. It’s been a fun couple of days!

My team creates software to manage 401(k) retirement plans. Back in 2006, the U.S. Congress passed legislation allowing Roth contributions in 401(k) plans. Plan participants can elect to pay taxes on their retirement deferrals up front, rather than the usual tax-deferred 401(k) contributions. Then, when they retire and withdraw their savings, they don’t have to pay taxes on their withdrawals. This is called a “qualified” distribution.

There is a catch – if you make a Roth contribution, you have to leave it in your account for at least 5 years. If you withdraw it early, you have to pay taxes on whatever gains you made from interest and dividends on your investments. We tested the difference between qualified and unqualified distributions both with in-memory FitNesse tests that verified the algorithms in the code, and by manual testing by faking out the dates as needed. But there was no chance yet to have a qualified distribution in real life.

We have many tests for Roth withdrawals, from the unit level on up to the GUI level. One FitNesse test verifies the results when an employee withdraws money that includes funds from their Roth account. Since 2006 was the first year anyone could make Roth contributions, there was no possibility to make a “qualified” distribution until this year. The test set up a participant whose first Roth contribution was in 2006, and verified that the system withheld taxes for the gain portion when the participant withdrew the money.

At the start of 2011, this test started to fail in our continuous integration. Surprise – it’s 2011, the participant has now been contributing Roth funds for 5 years, thus now the distribution is qualified and should on longer have any tax withheld! The actual results of the test showed this difference in behavior.

Some people would say we shouldn’t have an end-to-end test like this, that it’s expensive to maintain. But I think it was cool to see this functionality actually work in production. It allowed me to give a heads-up to the CSRs that they would now see Roth distribution checks going out that didn’t have tax withheld. I changed the expected results, because before this, we weren’t able to have any end-to-end test for this case. It also shows how our tests are living documentation – as soon as we have a test failure, we have to figure out the reason, and either change the code (if it’s a bug) or change the test (if it’s correct behavior).

What’s your opinion? Do you like this sort of test, and this sort of surprise?

My teammate Tony Sweets (sys admin, programmer and genius) just posted a terrific writeup on StickyMinds/Techwell of how our continuous integration and build process has evolved over the years. This includes a history of the hardware we’ve used and how the builds have been set up, both in CruiseControl and Hudson, and all the different things the build process does.

One of my favorite excerpts:

This method worked too well. The testers started splitting up their test suites and were creating new build jobs left and right mainly because it was so simple. They didn’t have to deal with hostname issues and whatnot. They simply had to create the new job from an existing Hudson job and change whatever they needed in order to run a different set of tests. When the job ran it did not care what else was running because it was running in its own separate environment.

I love the easy-to-use Hudson UI, and this freedom to configure our own test jobs the way we want. It’s also easy to stop jobs or kick them off when we want.

Whenever I speak to a conference session or user group meeting, I always tell people, “If you aren’t doing continuous integration now, go back to your office and drop everything and get your CI going. It isn’t hard to do, there are a bunch of good tools available to help, even Testify Wizard to help you set it up. A programmer can do it in a matter of days or less. There’s no excuse to not do CI.”

I’m convinced that in 5 years at the most, any team not doing CI will be looked upon the same way we look upon teams that don’t do source code control. It would just be crazy to not do it! Automated tests don’t have much value if they aren’t giving you quick feedback several times a day. Without CI, your technical debt is bound to bury you quickly.

If I had to pick one reason our team has been so successful the past 7 years, our CI process is it. It’s the pulse of our team, and if it stops (as it did a few weeks back – see Tony’s blog post!), we all just about have a heart attack! When it’s ticking along, we feel healthy and happy.