Agile Testing with Lisa Crispin

Yesterday I had a call from a participant in my recent BayAPLN session. I completely screwed up and, when I tried to save the phone message, I deleted it, and somehow the number that my phone saved, I could not call back. So, I’d like to try to answer the question, as best I remember it, here. If you still have questions, please email me, because this is such an important topic.

Since I listened to this question in my car and could not write it down, I don’t have it exactly, but it was along the lines of: “My team has no automated testing. One of our user stories did not have all the testing completed by the end of the sprint, because there was not time to complete the manual tests. Would you please recommend tools to help us?”

I know of a lot of great testing tools, GUI drivers and frameworks. If you want to do GUI testing, you can’t go wrong with Canoo WebTest, Selenium, Watir. Robot Framework, FitNesse and other frameworks can be used for testing at the API level or combined with a driver for testing through the GUI. Those are just a few examples of what’s available. Even better, your team can write its own testing frameworks and harnesses.

But tools aren’t the point. The point is that the best people to solve your testing problems is your own development team. By “development team”, I mean programmers, testers, BAs, DBAs, sys admins, everyone involved in delivering software. If you aren’t delivering user stories in a sprint because testing isn’t finished, you need to look at that problem as a team. By bringing all your team’s different roles, experiences and viewpoints to bear on this problem, you will think of experiments to try, and you will arrive at solutions.

Give yourselves time to experiment. Engage everyone involved in delivering software in solving the issues of getting all testing activities done each sprint or iteration. This diversity of viewpoints means you will eventually succeed. YOUR TEAM can solve this problem. Give it a chance!

On March 24, I facilitated a workshop at Agilistry Studios in Pleasanton, CA, on Learning for Testers. 19 practitioners from the Bay Area got together to consider how, as testers, we can learn ways to contribute more to our teams, as well as raising the bar for our profession. I’d like to share some of the ideas generated by this engaging group!

Why We Should Learn

We started out by considering good reasons that testers should learn. Participants self-organized into groups, brainstormed ideas, and each group chose their top three ideas to share. Some of the items surprised me. One reason to learn was “to reduce stress”. I asked the group to elaborate. They pointed out that it is stressful if you aren’t sure whether you have the appropriate skills to do your job. That is so true, I have felt that stress before!

Another category of reasons that resonated with me was “to have fun” and “not to be bored and have ADD”. Finding joy in our work should be our highest priority IMO!

Rather than take up room on this page, I invite you to see the photos of the actual results on my Picasa site.

What We Should Learn

I love the Bay Area, it is truly a crucible of software creativity. It is no surprise, therefore, that the ideas of what to learn were surprising to me.

“Knowing when to stop” – wow, so true. Any good tester can find things to test forever. When have we achieved the “minimum”? What is “enough”? Given that in real life, we have time and resource limitations, this is a crucial skill.

“How to invent test ideas” – cool!

“Deciding what is most important to test” – that kind of ties in with “Knowing when to stop”, I think. Knowing how to analyze risks, learning customer and user priorities, these are necessary skills for testers.

I was glad to see “thinking skills” were a priority, such as: Communication, Integrity, How to Teach. Integrity is particularly interesting to me. It’s essential, but how do we learn it? How do we teach it? Comments, please!

Another interesting skill to learn was the “Elevator Pitch”. If you can describe what you do in under a minute, you must really understand it. This exercise produced many intriguing skills – please check out the photos of the sticky notes.

How To Learn These Things?

I think the most innovative ideas in this workshop surfaced during the “How To” brainstorming.

Consider this: “Note your assumptions. Later, compare what happens to your assumptions. When you stop, note your reasons for stopping. Mindfulness.” I don’t know about you, but I am going to try this.

Journalism class, to learn how to make elevator pitches – what a great idea! We have to go outside our profession to get skills that will help us do our best work.

Risk taking – set out to fail! This takes a lot of courage, but failure is a great way to learn, so why not?

And finally, my favorite, “Learn by doing”. Isn’t that the best way? But to learn, we need time. If we’re always focused on meeting deadlines, we won’t absorb any learning.

Make some room today for learning. Your team has the best solutions to your problems. You just need to take the time to talk about them and think of small experiments (as Linda Rising says) to address them.

My colleague and co-author Janet Gregory and I have a new article in this month’s Better Software about Learning for Testers, with some additional material on Techwell.com. We’d love to know what you think, please comment!

My Software Quality Connection article, Selling Agile to the CFO, has been reformatted by ITBusinessEdge into a nice little slide deck. I think it is really easy to read & digest in this new format.

I have a new article on TechTarget’s SearchSoftwareQuality.com site, “Getting on the Same Page: How Testers Can Help Clarify Requirements”. I’d love to hear of your own experiences in this area.

Recently I wrote a post about how our team used a mind map to help us with theme planning. We also sometimes use mind maps to help us in planning our testing. Some people asked if I’d post an example. Here it is:

Theme Testing Mind Map

We’re a financial services company, and this theme was a rewrite of one of the oldest and most critical parts of our system: sweeping a tiny percentage of the assets in each participant’s account as pay for our services.

My fellow tester Lori Thayer (who came up with the idea to mind map test cases) and I wrote the initial mind map, then discussed it with the rest of the development team, the product owner, the controller and other stakeholders. Our central node is “Run”, because this is a batch process that needs a way to be kicked off, make sure prerequisites are in place, provide a way to monitor progress, and be restartable in case of problems. Not only is it important to test these parts of the system, but they’ll be invaluable to help us execute and monitor tests.

Without boring you with all the details of this them, you can see that the map includes some primary nodes such as “collection” for sweeping shares out of the account, “sale” for selling the shares once they’re in our company account (definitely in our best interest!), and “instructions”, each representing one participant’s account. We drew dotted lines between related nodes, such as “collection” and “sale”, “sale” and “fast401k account”. The map includes details such as the internal account number where the funds should end up. On the right side, we wrote questions that nobody was able to answer yet. These got answered as we learned more through talking to stakeholders and doing exploratory testing on a design spike.

We referred to this map many times over the next several iterations as we implemented this theme. We checked off the areas that were tested, added more questions and/or answers, even added more nodes as we thought of more areas to test. We didn’t get down into level of test case detail, but the map shows the relationships we needed to test, in addition to the individual components. (Unfortunately, I neglected to take an “after” photo).

We got into more details on the team wiki, using color-coding to show questions and answers. Our most senior programmer lives in India, and the wiki is a good way for him to participate in discussions like this.

Wiki Page Example

Here’s a snippet of some test scenarios we wrote. When we’re satisfied with a test, we put our initials next to it to indicate it’s done.

Example Test Scenarios

It took us five iterations (ten weeks total) to finish this theme. At the end, we felt confident that we’d thought of and discussed all angles of the process, its potential impacts on other parts of the system,  talked to all stakeholders about their individual needs and concerns, and done adequate testing not only at the functional level, but also performance and usability.

Agile is pretty mainstream now, but some folks still aren’t sure what testers do. In addition, the “Whole Team” approach to quality and the idea that testing and coding aren’t separate phases still mystifies some people. I think my team’s activities the past two days are a good illustration of how agile teams work together to produce high quality software, yet we testers still make valued contributions.

Our software manages all aspects of 401(k) retirement plans. A “plan sponsor” is the company or employer that establishes the 401(k) plan for their employees. We’re working on a theme to allow more than one person working at a company log in and do “plan sponsor” functions for their 401(k) plans, such as submitting payroll contributions or enrolling employees.

Up to now, since only one “plan sponsor” account existed per 401(k) plan, we called that user role “plan sponsor”. This term appears all over our website. Now, we are working on a theme to allow multiple accounts to do the “plan sponsor” functions. These users are not plan sponsors – the employer is. Earlier this week, we had a couple of different discussions on what to call these users and what terminology to use on the user interface. Every manager and developer had a different idea, we kept going round and round discussing it in twos and threes.

Yesterday, I scheduled a meeting with ALL the stakeholders to make a final decision on terminology. You might say, “It’s not the tester’s job to call a meeting like that! Shouldn’t the ScrumMaster or Product Owner do that?” Well, they could, but our ScrumMaster was out sick, and our PO is always swamped. We needed to stop wasting time changing verbiage over and over. In 15 minutes, the various business managers agreed on the term “plan administrator” and on some other related terminology.

Changing terminology - from a mockup

Since the PO was too busy to do it, I mocked up all the changes for all the various pages, posted them on the wiki for our remote developer to see along with narrative requirements and test cases for the changes, and stuck them to our task board along with a task card. Our remote developer, who lives many time zones ahead of us, made most of the changes before we got to work the next morning, and another developer made the rest. I verified the changes, showed the PO, and updated an automated regression test.

We had an issue, though. The modules and classes in our code still used the “plan sponsor” term. We have a “plan_sponsor” table in the database and a column called “sponsor_id”. Eventually it would become very confusing that our UI says “plan administrator” all over the place, and the code has “plan sponsor”. One of our developers decided to copy the “plan_sponsor_mgmt” module to a new module called “plan_admin_mgmt”, and renamed anything in the new module with “plan sponsor” references to be “plan admin” references. This required renaming classes, config files, variables and more. He deleted the code from the old module and will delete the old module itself when the dust settles. (He found out later he could have done the renaming in the old module, but as he says, live and learn).

Later, we will change URLs, velocity variables and spring field declarations, but as that will change HTML Ids and names and cause automated GUI regression scripts to fail, we are coordinating that. And, we still need to change the database table and column names.

It’s a lot of trouble, but it would have been more trouble later, when someone wants to change the “plan administrator” functionality, and can only find “plan sponsor” in the code. We all think we’ll remember everything forever, but we have a big code base considering our team size, and what if we hire a new developer?

So what’s my point? As a team, we did what needed to be done to maintain a high standard of quality, and avoid technical debt. No manager told us to do that. Our Product Owner did not “prioritize” refactoring the code to have the new terminology, because that isn’t a business decision – we control our internal code quality. Though everyone on our team cares about quality, I made my own particular contribution by getting everyone to agree on terminology and verbiage, writing a task card and posting the mockups. It’s been a fun couple of days!

I report on my team’s experiences with sizing and estimating themes and stories in an article just published on StickyMinds, Estimation Roller Coaster. I’ve had some good feedback already via Twitter. Please let me know what you think!

Today was day 2 of a new sprint. We’re working on a big, complex theme. Up to now, our system allowed only one employer or “plan sponsor” account for every 401(k) retirement plan. Many companies want several plan sponsor accounts: one for their accountant who submits the payroll contributions, one for the human resources manager who enrolls employees, one for the company lawyer who reviews the legal documents, and so on. With this theme, we’re making it possible for each plan to have multiple sponsor accounts.

We created a mind map during our theme planning and it shows that this change affects many parts of our system. We identified several user stories, but felt there were still some loose ends we hadn’t yet identified. We wrote a story called “Close the gap” and wrote one big development task card with 32 hours on it. We wrote lots of testing cards, to identify the “gaps” and do lots of regression and exploratory testing.

Friday and today, I explored the system looking for those gaps. Last sprint, I had already discovered several reports that didn’t work for the additional sponsor accounts, so I went looking for more reports. I logged on as all the different user types in our system: customer support reps, plan administrators, third-party administrators, plan advisors, as well as plan sponsors and employees (participants). I found more problematic reports and UI pages, and some searches that didn’t work.

Here’s an example of what I learned. We have an “account details” page that is used by three different roles that shows information about a given 401(k) plan. It links to an “account summary” report that lists all the information about the plan. Both the details page and the summary report list the plan sponsor for that plan. Now that a plan could have more than one plan sponsor, what should show in these locations?

I went and talked to the VPs over four different departments of our small company that have a stake in this functionality, each with a different point of view. We looked at the account details page and the account summary report, and agreed that a simple approach would be best: change the label “Sponsor” on the account details page to “Account Representative”, a new concept in our system which identifies the person whose name goes on all the legal documents. In addition, we’d have a new page that lists all the sponsors for the plan, with columns for each item of information such as phone number and email, and link to that new page from the account details page. We can probably use that same page in other areas of the application. The sponsor information can be removed from the account summary report.

I discussed this idea with one of the developers, who agreed it was probably the simplest and least costly way to provide the information. Then I documented it on the wiki page for the “Close the Gaps” story, including screenshots on which I made notes about what to do. I wrote high-level test cases, and identified where we need more detailed, automated test cases (I’ll start on those when a developer starts working on the coding tasks).

The Product Owner thanked me for finding some unexpected places where this theme “poked out”, and helping to identify low-cost solutions. I enjoyed exploring some of the more obscure parts of our application, learning more about how they work and what needs to be changed, and writing specifications and test cases so that the developers will know what to code.

Here’s a snippet of the wiki page. The links bring up annotated screenshots. (Click the image so you can read it!)

part of Wiki page for Close the Gaps story

My team creates software to manage 401(k) retirement plans. Back in 2006, the U.S. Congress passed legislation allowing Roth contributions in 401(k) plans. Plan participants can elect to pay taxes on their retirement deferrals up front, rather than the usual tax-deferred 401(k) contributions. Then, when they retire and withdraw their savings, they don’t have to pay taxes on their withdrawals. This is called a “qualified” distribution.

There is a catch – if you make a Roth contribution, you have to leave it in your account for at least 5 years. If you withdraw it early, you have to pay taxes on whatever gains you made from interest and dividends on your investments. We tested the difference between qualified and unqualified distributions both with in-memory FitNesse tests that verified the algorithms in the code, and by manual testing by faking out the dates as needed. But there was no chance yet to have a qualified distribution in real life.

We have many tests for Roth withdrawals, from the unit level on up to the GUI level. One FitNesse test verifies the results when an employee withdraws money that includes funds from their Roth account. Since 2006 was the first year anyone could make Roth contributions, there was no possibility to make a “qualified” distribution until this year. The test set up a participant whose first Roth contribution was in 2006, and verified that the system withheld taxes for the gain portion when the participant withdrew the money.

At the start of 2011, this test started to fail in our continuous integration. Surprise – it’s 2011, the participant has now been contributing Roth funds for 5 years, thus now the distribution is qualified and should on longer have any tax withheld! The actual results of the test showed this difference in behavior.

Some people would say we shouldn’t have an end-to-end test like this, that it’s expensive to maintain. But I think it was cool to see this functionality actually work in production. It allowed me to give a heads-up to the CSRs that they would now see Roth distribution checks going out that didn’t have tax withheld. I changed the expected results, because before this, we weren’t able to have any end-to-end test for this case. It also shows how our tests are living documentation – as soon as we have a test failure, we have to figure out the reason, and either change the code (if it’s a bug) or change the test (if it’s correct behavior).

What’s your opinion? Do you like this sort of test, and this sort of surprise?

I am honored that Agile Testing: A Practical Guide for Testers and Agile Teams by me and Janet Gregory is on Jonas Bandi’s list of his favorite software development books of 2010, especially because the other books in the list are also some of my favorites for the year. The start of a new year is a good time to set learning goals, and this list of books would make a great place to start.

I have a new blog post on my Stickyminds blog with my thoughts as I am in the middle of a big, complex, high-risk project. This has been a highly unusual project for our team, which just goes to show, even after 7 years as an agile team, new challenges constantly present themselves. Please share your own project experiences in the comments.