Team Commitment to Quality

There are so many test frameworks, drivers and other tools out there I want to try. For example, Selenesse has been at the top of my list of things to try for months (and I’ll get to it sometime soon!) However, I couldn’t resist the offer from Pekka Klarck, one of the Robot Framework contributors, to give me a live one-on-one demo of Robot Framework a couple of weeks ago. This weekend, I finally had time to try to write my first test (really, to finish up a test that Pekka had started for me).

My first test is below. I always struggle mightily when I learn some new tool or language, and this was no exception. (Apparently my more-geeky friends also struggle with new things, but they don’t whine publicly about it like I do). The user doc for RF leaves a bit to be desired from a non-programmer perspective. However, they are actively improving it. With much help from Pekka (and suggestions from many in the Twitterverse who also use RF), I have gotten over this first beginner hump, I can see lots of possibilities.

You can write RF tests in various formats, including HTML tables. I like the plain text format, because to me, simple is good. It makes the tests more amenable to version control, because you can easily diff versions to see what has changed. If you try that with html table tests you’ll have lots of noise.

I love the flexibility of this tool. I am trying it out with the RF Selenium libraries, but you can use it with a variety of tools, including tools to test non-Web apps such as Java Swing. There’s a lot of potential there.

I like the fact that you can define whatever “keywords” you want, so you can create your own domain-specific language and the test can document itself and the code that it is testing. The inline-documentation aspect is also one of my favorite features of FitNesse.

My team is pretty happy with our current tools: FitNesse for testing behind the GUI, Canoo WebTest for GUI regression tests, and Ruby/Watir scripts to help with exploratory testing. We’ve long wanted to try Selenium, though, and RF would give us an easy way to do that. We’re always experimenting, and an experiment with RF will tell us whether there’s value in adopting it.

For myself, I think RF will be useful in my public presentations and classes. I’m always looking for ways to teach good test automation design, and I think I can do this fairly easily with RF. I’ve seen Dale Emery and Elisabeth Hendrickson do good demos with RF. I’ve also seen Antony Marcano, Andy Palmer and Gokjo Adzic do good demos with FitNesse, so I’m not saying one is better for this purpose than the other!

Here’s my test. I also copied it in below, though the spacing might be wacky there. I put comments in for myself that might help you, too. Download the RF-Selenium library demo and try it for yourself. If you have any issues I am glad to try to help. It would help me learn, too!


=”color: #333399;”>Developer responsible for story should think about all task cards ahead of time, make sure all necessary cards are there

  • Ensure everyone needed is copied on emails
  • Stop checking in untested code
  • We focused on finishing all the “leftovers” first. For the new FitNesse fixtures, the developers asked that I first write example tests with my idea of how the test should work. The remote developer then wrote the fixtures and updated the tests as needed so that they pass. These fixtures were difficult to do as they had to use a lot of legacy code, but the ROI will be big because they test critical areas where we’ve had expensive production problems in the past.

    There was only one incident of untested code being checked in, and we talked about it right away and discussed ways to ensure this stops happening.

    6 days into the 2 week sprint, we still had lots of testing cards and not many development ones. The developers, of their own volition, decided during the Scrum to attack the test cards. By the end of the day, most of the test cards were finished! Then they went on to the stories that were new that sprint.

    Since a glance at the online storyboard now tells us who the main developer is on each story, communication has improved. When more than one developer works on tasks for a story, they’re communicating better. Lots of pairing is happening too. Communication is better all around, including with the remote team member.

    8 days into the sprint, a developer was writing up the information on the functionality he had changed for a batch processing story, and as he wrote, he realized that he missed a use case. He fixed it, but was concerned there wasn’t enough time left in the sprint to adequately test the new functionality. We decided to postpone that story to the next sprint. Consciously deciding to put off testing to the next sprint isn’t a bad thing, when there’s a good reason for it.

    Although we are still very busy, it’s a great feeling to see our storyboard for the sprint with almost all the cards moved to the ‘done’ column. Since it’s holiday time, people are taking time off, so time is short. A task to automate testing for a new UI feature implemented with Ajax has been rolled over to the next sprint, because the test script is blowing up (anyone out there have experience with the mouseover command in WebTest?) Again, this was a conscious decision, not a case of “Oops, today is the last day of the sprint and we didn’t get this finished”.

    Many teams struggle with trying to finish all the testing for all the stories by the end of each iteration. If this happens to you, the first action to take is to limit the amount of new work brought into the next sprint, so you can focus on finishing the leftovers. As you catch up and are able to reduce your technical debt by finishing testing and test automation tasks, you’ll be able to increase your velocity later, without sacrificing quality. This has to be a team effort.

    Please post comments and questions and I’ll follow up on those. This is such a central factor to team and agile testing success, I’d like to talk about it more.

    13 comments on “Team Commitment to Quality

    1. While reading through Cockburn’s book on use cases, it really strikes me that we should be able to write down use cases as test cases. I would apply some tidying to achieve that in your example:

      type in username invalid
      would become
      enter username “invalid”

      type in password xxx
      would become
      with password “xxx”

      click submit
      verify the invalid account error message

      The last two I would combine to
      rejects login with “Invalid user name and/or password”

      I assume in the next few tests I would like to apply some method extractions, as the interface gets more clear, but I refuse to apply this up-front without particular contextual information at hand.

    2. Thanks, Markus, I will apply those changes. I do plan to extract out duplicate ‘code’, preferably into separate files, as I add to this. I wish I had a better sample app. Of course I can use my company’s app, but it doesn’t make a very good generic example to show other people.

    3. Thanks a lot for writing this blog post about Robot Framework and for a positive initial review. I agree that the entry level documentation is in bad shape and needs to be improved. I just started updating the SeleniumLibrary demo [1] and plan to create some very simple “first tests” that can be distributed with the core framework too. Help in creating examples and improving documentation is very highly appreciated!

      [1] http://code.google.com/p/robotframework-seleniumlibrary/issues/detail?id=106

    4. Hi Lisa,

      A collegue of mine attended one of your tutorials at StarEast and was very impressed 🙂

      We are in the process of implementing an agile testing methodology on a project and I was wondering if you could share some high level info regarding your current project? I find real world examples typically have more impact in my organization 🙂

      If this is something that can be shared I would love to know@

      Number of Fitnesse tests?
      Number of Canoo Web tests?
      Amount of time spent on manual testing?

      % testing you guys perform via the API?
      % of testing you guys perform via the front end?

    5. Well, that’s a big question and it’d end up being a whole blog post at least. But the quick answer is we have about 4500 JUnit tests (each having multiple assertions), 550 FitNesse test pages (with tens of thousands of test steps), about 13000 Canoo WebTest test steps, and 1650 Watir test assertions. We have done a pretty good job of getting a right-side-up automated test pyramid.

    6. thanks, I really appreciate the info.

      I think you highlight an important point – that for really good testing you do need to hit the application from different angles – and you are using several tools to do that via testing the java classes, the application API, the web front end, etc

      Apologies for getting waaaay off topic but I’m curious to know how/ why you are using Watir for exploratory testing? I always thought exploratory testing was more a “manual testing’ discipline

    7. I am newbie to robot framework.Can you please provide some doc& example

      how to use with selenium to test java application.

      Thanks in advance.

    8. hi i need your help regarding the folllowing code,could u pls help me.
      You should make a python program with the following spec.
      – full python, no use of python library except the one in the official python installation and Robotframework
      – a main program (M1) start a thread T2 and then start a loop where it reads messages from T2
      – the thread T2 start robotframework, to execute a robotframework test.

      The test is a small sequence
      print xx
      print xy
      print xz

      Each pause must generate an exchange between T2 and M1. The exchange is T2 send a message (with Queue python module) to M1. In M1 the reading message loop, prints a text, and send a message back to T2.
      After having send a message to M1, T2 was waiting from the message back (on another Queue). When it gets the message, that is the end of the pause instruction implementation

      The pause instruction is implemented in python

    Leave a Reply

    Your email address will not be published. Required fields are marked *


    Recent Posts: