Agile Testing with Lisa Crispin

I work for a mid-sized software company with around 27 development teams, mostly Scrum but also a few doing Lean and Kanban. We have a large and diverse product of HR-related software, I think there are around 6 million LOC, on three different platforms, including an ASP product and a .NET product. I’m fairly new, I started at the beginning of June, so I’m still learning about how everything gets done. We release four times a year and the software has won many prestigious awards. I’ve been impressed with this large-scale implementation of agile development. Just the speed of the daily Scrum of Scrums is amazing.

Many of the teams do a great job of automating their tests, and a few teams have their own continuous integration (CI) that runs multiple times a day. Other teams have automated tests, but must wait for the company-wide build to run, which is usually only once per day and sometimes not that often. They don’t get feedback fast enough, and that slows them down – a lot.

I’ve been asking why some teams struggle to automate tests, and why some teams don’t have their own CI. I’ve also asked my own team why most of our functional tests are automated through the GUI, rather than behind the GUI. We use FitNesse, but mainly to drive GUI tests with SWAT.

The answer I get a lot is “It’s the framework”. We have three architecture teams, and there is a framework (maybe there is more than one, I don’t fully understand it yet) that apparently makes it hard to test at the page or object layer. In addition, some teams’ code has tentacles into legacy code written in Delphi whose tests are written in WinRunner. Also, the current framework appears to not be fully supported with automated unit tests, it was developed a few years back. Programmers new to the framework seem to have a hard time understanding how to work with it. The framework clearly causes a lot of frustration. A new framework is under development, but it won’t be available for some time.

In order to understand the existing framework and find out whether the new one will facilitate testing behind the GUI, I was fortunate to get to meet with the leader of the architecture side of the development group. He gave me an overview of the new framework’s current design (which may change), aimed at making our product easily configurable by customers, using web services. It looks like this design allows testing under the GUI level.

Additionally, I learned that it is possible to test under the GUI in the current framework, and I met someone on another team who has automated many FitNesse tests at the object level. I plan to get her to show me how these tests work, and see if what she did would work for our code.

I haven’t worked at a company this large since the early 90s, and I’ve never worked in a company, agile or otherwise, that had “architecture teams” and “frameworks”. I get it that our product needs to look and work consistently across the board, so it does make some sense to centralize the design of the UI and perhaps other parts of the system. However, if the architecture is done by one team, what does that mean for programmers on other teams? Is the fun part of their job taken away? Do they get a chance to think for themselves and improve the product?

It also seems to me that there needs to be a lot of communication amongst the entire development community of our company, that all programmers should have some input to the architecture process. I’ve found that communication between teams in general is a bit lacking. Each team is so focused on delivering its part of the product, they don’t always feel they have time to go talk to other teams. So, for example, perhaps nobody on my team has talked to the person who did a lot of FitNesse functional tests.

Does your company have an architecture team, and/or a framework that everyone works with? Please comment and share your experiences. As a tester, I’d like to know what I can do to help my team deliver the highest possible quality when we have to work with a framework designed by someone else.

Weekend before last I had the great pleasure of participating in the Austin Workshop for Test Automation. Test tool developers and users exchanged experiences and ideas. You know that automated testing has come a long way when you see the approaches organizations such as Opera and Expedia use to overcome massive testing challenges.

One discussion that helped a penny drop in my brain was the concept of test libraries vs. test frameworks. A library is a reusable component. Watir is a family of Ruby libraries that drive the browser. A framework is a set of libraries that gives you everything you need to test. Cucumber is a test framework that supports behavior-driven development using plain text. Taza is a test framework that uses the Watir libraries and generates RSpec test assertions. At least – I think so! Please correct me if I’m wrong.

A good analogy someone came up with at AWTA is that Kevin O’Connor’s (host of This Old House) tool shop is a framework – it has everything needed for any project, whereas Marek J‘s garage is just a bunch of tools. This is not to cast aspersion on Marek J’s own cool test automation framework, Watirloo.

What I learned is that my team has our own test automation framework using Watir, which consists of our Ruby scripts and test/unit. The BDD frameworks such as RSpec and Cucumber seem to be the cooler thing to use than test/unit, but I don’t know how we would take on the huge task of refactoring all our Ruby/Watir scripts to use a different test framework.

That doesn’t mean we shouldn’t look at some of these newer frameworks to see if they have something to offer that our current automation solutions don’t include. My fellow tester Lori Bergan has been looking into Cucumber. I’ve been intrigued by BDD since last year’s CITCON Denver. I like tools that seem natural.

What tools have you been trying lately? Are you happy with the ones you currently use?