Agile Testing with Lisa Crispin

Recently I wrote a post about how our team used a mind map to help us with theme planning. We also sometimes use mind maps to help us in planning our testing. Some people asked if I’d post an example. Here it is:

Theme Testing Mind Map

We’re a financial services company, and this theme was a rewrite of one of the oldest and most critical parts of our system: sweeping a tiny percentage of the assets in each participant’s account as pay for our services.

My fellow tester Lori Thayer (who came up with the idea to mind map test cases) and I wrote the initial mind map, then discussed it with the rest of the development team, the product owner, the controller and other stakeholders. Our central node is “Run”, because this is a batch process that needs a way to be kicked off, make sure prerequisites are in place, provide a way to monitor progress, and be restartable in case of problems. Not only is it important to test these parts of the system, but they’ll be invaluable to help us execute and monitor tests.

Without boring you with all the details of this them, you can see that the map includes some primary nodes such as “collection” for sweeping shares out of the account, “sale” for selling the shares once they’re in our company account (definitely in our best interest!), and “instructions”, each representing one participant’s account. We drew dotted lines between related nodes, such as “collection” and “sale”, “sale” and “fast401k account”. The map includes details such as the internal account number where the funds should end up. On the right side, we wrote questions that nobody was able to answer yet. These got answered as we learned more through talking to stakeholders and doing exploratory testing on a design spike.

We referred to this map many times over the next several iterations as we implemented this theme. We checked off the areas that were tested, added more questions and/or answers, even added more nodes as we thought of more areas to test. We didn’t get down into level of test case detail, but the map shows the relationships we needed to test, in addition to the individual components. (Unfortunately, I neglected to take an “after” photo).

We got into more details on the team wiki, using color-coding to show questions and answers. Our most senior programmer lives in India, and the wiki is a good way for him to participate in discussions like this.

Wiki Page Example

Here’s a snippet of some test scenarios we wrote. When we’re satisfied with a test, we put our initials next to it to indicate it’s done.

Example Test Scenarios

It took us five iterations (ten weeks total) to finish this theme. At the end, we felt confident that we’d thought of and discussed all angles of the process, its potential impacts on other parts of the system,  talked to all stakeholders about their individual needs and concerns, and done adequate testing not only at the functional level, but also performance and usability.

There are so many test frameworks, drivers and other tools out there I want to try. For example, Selenesse has been at the top of my list of things to try for months (and I’ll get to it sometime soon!) However, I couldn’t resist the offer from Pekka Klarck, one of the Robot Framework contributors, to give me a live one-on-one demo of Robot Framework a couple of weeks ago. This weekend, I finally had time to try to write my first test (really, to finish up a test that Pekka had started for me).

My first test is below. I always struggle mightily when I learn some new tool or language, and this was no exception. (Apparently my more-geeky friends also struggle with new things, but they don’t whine publicly about it like I do). The user doc for RF leaves a bit to be desired from a non-programmer perspective. However, they are actively improving it. With much help from Pekka (and suggestions from many in the Twitterverse who also use RF), I have gotten over this first beginner hump, I can see lots of possibilities.

You can write RF tests in various formats, including HTML tables. I like the plain text format, because to me, simple is good. It makes the tests more amenable to version control, because you can easily diff versions to see what has changed. If you try that with html table tests you’ll have lots of noise.

I love the flexibility of this tool. I am trying it out with the RF Selenium libraries, but you can use it with a variety of tools, including tools to test non-Web apps such as Java Swing. There’s a lot of potential there.

I like the fact that you can define whatever “keywords” you want, so you can create your own domain-specific language and the test can document itself and the code that it is testing. The inline-documentation aspect is also one of my favorite features of FitNesse.

My team is pretty happy with our current tools: FitNesse for testing behind the GUI, Canoo WebTest for GUI regression tests, and Ruby/Watir scripts to help with exploratory testing. We’ve long wanted to try Selenium, though, and RF would give us an easy way to do that. We’re always experimenting, and an experiment with RF will tell us whether there’s value in adopting it.

For myself, I think RF will be useful in my public presentations and classes. I’m always looking for ways to teach good test automation design, and I think I can do this fairly easily with RF. I’ve seen Dale Emery and Elisabeth Hendrickson do good demos with RF. I’ve also seen Antony Marcano, Andy Palmer and Gokjo Adzic do good demos with FitNesse, so I’m not saying one is better for this purpose than the other!

Here’s my test. I also copied it in below, though the spacing might be wacky there. I put comments in for myself that might help you, too. Download the RF-Selenium library demo and try it for yourself. If you have any issues I am glad to try to help. It would help me learn, too!

—-

lisas_first_test.txt

# new line and 2 spaces are cell delimiters.
# the asterisks have to start in the first col
# Look at the selenium keywords in http://code.google.com/p/robotframework-seleniumlibrary/wiki/LibraryDocumentation
# Look at the quick start guide for more http://code.google.com/p/robotframework/wiki/QuickStartGuide
Actually you don’t even need the # if you type up above the first table. Tables can be in any order. ‘*** Test Cases***’ is an example of a table.
Run it with ./rundemo.py lisas_first_test.txt
To start the server for the app under test manually, python httpyserver.py start

*** Test Cases ***

invalid account
navigate to the login page
type in username  invalid
type in password  xxx
click submit
verify the invalid account error message

*** Keywords ***

navigate to the login page
log  hello, world!
open browser  ${LOGIN PAGE}  ${BROWSER}
title should be  Login Page
${title} =  Get Title
should start with  ${title}  Login

type in username  [Arguments]  ${username}
input text  username_field  ${username}

type in password  [Arguments]  ${password}
input text  password_field  ${password}

click submit
Click Button  login_button

verify the invalid account error message
Title should be  Error Page
Page should contain  Invalid user name and/or password

*** Settings ***
Library  SeleniumLibrary
Test Teardown     close browser

*** Variable ***
${LOGIN PAGE}   http://localhost:7272
${BROWSER}      firefox