Agile Testing with Lisa Crispin

There are so many test frameworks, drivers and other tools out there I want to try. For example, Selenesse has been at the top of my list of things to try for months (and I’ll get to it sometime soon!) However, I couldn’t resist the offer from Pekka Klarck, one of the Robot Framework contributors, to give me a live one-on-one demo of Robot Framework a couple of weeks ago. This weekend, I finally had time to try to write my first test (really, to finish up a test that Pekka had started for me).

My first test is below. I always struggle mightily when I learn some new tool or language, and this was no exception. (Apparently my more-geeky friends also struggle with new things, but they don’t whine publicly about it like I do). The user doc for RF leaves a bit to be desired from a non-programmer perspective. However, they are actively improving it. With much help from Pekka (and suggestions from many in the Twitterverse who also use RF), I have gotten over this first beginner hump, I can see lots of possibilities.

You can write RF tests in various formats, including HTML tables. I like the plain text format, because to me, simple is good. It makes the tests more amenable to version control, because you can easily diff versions to see what has changed. If you try that with html table tests you’ll have lots of noise.

I love the flexibility of this tool. I am trying it out with the RF Selenium libraries, but you can use it with a variety of tools, including tools to test non-Web apps such as Java Swing. There’s a lot of potential there.

I like the fact that you can define whatever “keywords” you want, so you can create your own domain-specific language and the test can document itself and the code that it is testing. The inline-documentation aspect is also one of my favorite features of FitNesse.

My team is pretty happy with our current tools: FitNesse for testing behind the GUI, Canoo WebTest for GUI regression tests, and Ruby/Watir scripts to help with exploratory testing. We’ve long wanted to try Selenium, though, and RF would give us an easy way to do that. We’re always experimenting, and an experiment with RF will tell us whether there’s value in adopting it.

For myself, I think RF will be useful in my public presentations and classes. I’m always looking for ways to teach good test automation design, and I think I can do this fairly easily with RF. I’ve seen Dale Emery and Elisabeth Hendrickson do good demos with RF. I’ve also seen Antony Marcano, Andy Palmer and Gokjo Adzic do good demos with FitNesse, so I’m not saying one is better for this purpose than the other!

Here’s my test. I also copied it in below, though the spacing might be wacky there. I put comments in for myself that might help you, too. Download the RF-Selenium library demo and try it for yourself. If you have any issues I am glad to try to help. It would help me learn, too!

—-

lisas_first_test.txt

# new line and 2 spaces are cell delimiters.
# the asterisks have to start in the first col
# Look at the selenium keywords in http://code.google.com/p/robotframework-seleniumlibrary/wiki/LibraryDocumentation
# Look at the quick start guide for more http://code.google.com/p/robotframework/wiki/QuickStartGuide
Actually you don’t even need the # if you type up above the first table. Tables can be in any order. ‘*** Test Cases***’ is an example of a table.
Run it with ./rundemo.py lisas_first_test.txt
To start the server for the app under test manually, python httpyserver.py start

*** Test Cases ***

invalid account
navigate to the login page
type in username  invalid
type in password  xxx
click submit
verify the invalid account error message

*** Keywords ***

navigate to the login page
log  hello, world!
open browser  ${LOGIN PAGE}  ${BROWSER}
title should be  Login Page
${title} =  Get Title
should start with  ${title}  Login

type in username  [Arguments]  ${username}
input text  username_field  ${username}

type in password  [Arguments]  ${password}
input text  password_field  ${password}

click submit
Click Button  login_button

verify the invalid account error message
Title should be  Error Page
Page should contain  Invalid user name and/or password

*** Settings ***
Library  SeleniumLibrary
Test Teardown     close browser

*** Variable ***
${LOGIN PAGE}   http://localhost:7272
${BROWSER}      firefox

There’s so much swirling around in my brain from a whole week in Chicago at Agile 2009. Let me start with my recollections of the Agile Alliance Functional Test Tools workshop. Jennitta Andrea chairs this effort, and Elisabeth Hendrickson facilitated this meeting.

The facility was great, plenty of room and good connectivity, and had a nice view of the Sears (Willis) Tower and a sliver of lake!

The workshop was all Open Space, so we started out by planning the sessions. 3 timeslots, 4 locations, though there ended up being only 11 sessions. One of the first sessions was lightning talk demos of various tools – Canoo WebTest, Twist, Cucumber, Robot Framework and SWAT. (Not to say that Watir and Selenium were ignored – we were just already familiar with those). I was impressed by Robot Framework, an open source tool whose developer is Pekka Klark. It has a lot of flexibility. It uses a tabular format similar to FitNesse except there is only one type of table. You can do keyword driven, data driven and even BDD style tests. It also takes command line arguments. It lets you have internal and external libraries such as Swing.

Mike Longin (who works for the same company I do) did a great job demo-ing both SWAT and UltiFit (the latter is not open source, but as there was interest in it, Mike may undertake that task). There was a lot of interest in SWAT, nobody had seen it before and I think they were particularly impressed by the IDE.

One of the other morning sessions was how to get Cucumber to work with .Net. Of course you can run Cucumber tests on the browser no matter what your app language, but if you want to test in the same process as your app code, you need some bridge. After a lot of discussion of things like Iron Ruby, someone had the idea to use the Slim protocol between Cucumber and .Net. Aslak Hellesoy, the Cucumber developer, thinks this idea may have legs. I have some photos of the diagrams they drew and will upload them here when I get time.

Paul King, one of the WebTest contributors as well as the main Groovy developer, proposed doing more mixing and matching of testing frameworks/drivers/utilities/tools. He had a nice graphic showing what is available. We all agreed there are enough test runners, and the developer community should focus on solving new problems. Gerard Meszaros created a spreadsheet showing all the tools along with the features and requirements people might have for their test framework/drivers. Some of us got together in an afternoon session to tweak the tools spreadsheet and Gerard posted it on Google docs so that tool developers/users can add to it and fill it out. This is the start of a great resource both for users looking for the right tool to developers who want to know what they might be able to build on or adapt.

Weekend before last I had the great pleasure of participating in the Austin Workshop for Test Automation. Test tool developers and users exchanged experiences and ideas. You know that automated testing has come a long way when you see the approaches organizations such as Opera and Expedia use to overcome massive testing challenges.

One discussion that helped a penny drop in my brain was the concept of test libraries vs. test frameworks. A library is a reusable component. Watir is a family of Ruby libraries that drive the browser. A framework is a set of libraries that gives you everything you need to test. Cucumber is a test framework that supports behavior-driven development using plain text. Taza is a test framework that uses the Watir libraries and generates RSpec test assertions. At least – I think so! Please correct me if I’m wrong.

A good analogy someone came up with at AWTA is that Kevin O’Connor’s (host of This Old House) tool shop is a framework – it has everything needed for any project, whereas Marek J‘s garage is just a bunch of tools. This is not to cast aspersion on Marek J’s own cool test automation framework, Watirloo.

What I learned is that my team has our own test automation framework using Watir, which consists of our Ruby scripts and test/unit. The BDD frameworks such as RSpec and Cucumber seem to be the cooler thing to use than test/unit, but I don’t know how we would take on the huge task of refactoring all our Ruby/Watir scripts to use a different test framework.

That doesn’t mean we shouldn’t look at some of these newer frameworks to see if they have something to offer that our current automation solutions don’t include. My fellow tester Lori Bergan has been looking into Cucumber. I’ve been intrigued by BDD since last year’s CITCON Denver. I like tools that seem natural.

What tools have you been trying lately? Are you happy with the ones you currently use?