Agile Testing with Lisa Crispin

I work for a mid-sized software company with around 27 development teams, mostly Scrum but also a few doing Lean and Kanban. We have a large and diverse product of HR-related software, I think there are around 6 million LOC, on three different platforms, including an ASP product and a .NET product. I’m fairly new, I started at the beginning of June, so I’m still learning about how everything gets done. We release four times a year and the software has won many prestigious awards. I’ve been impressed with this large-scale implementation of agile development. Just the speed of the daily Scrum of Scrums is amazing.

Many of the teams do a great job of automating their tests, and a few teams have their own continuous integration (CI) that runs multiple times a day. Other teams have automated tests, but must wait for the company-wide build to run, which is usually only once per day and sometimes not that often. They don’t get feedback fast enough, and that slows them down – a lot.

I’ve been asking why some teams struggle to automate tests, and why some teams don’t have their own CI. I’ve also asked my own team why most of our functional tests are automated through the GUI, rather than behind the GUI. We use FitNesse, but mainly to drive GUI tests with SWAT.

The answer I get a lot is “It’s the framework”. We have three architecture teams, and there is a framework (maybe there is more than one, I don’t fully understand it yet) that apparently makes it hard to test at the page or object layer. In addition, some teams’ code has tentacles into legacy code written in Delphi whose tests are written in WinRunner. Also, the current framework appears to not be fully supported with automated unit tests, it was developed a few years back. Programmers new to the framework seem to have a hard time understanding how to work with it. The framework clearly causes a lot of frustration. A new framework is under development, but it won’t be available for some time.

In order to understand the existing framework and find out whether the new one will facilitate testing behind the GUI, I was fortunate to get to meet with the leader of the architecture side of the development group. He gave me an overview of the new framework’s current design (which may change), aimed at making our product easily configurable by customers, using web services. It looks like this design allows testing under the GUI level.

Additionally, I learned that it is possible to test under the GUI in the current framework, and I met someone on another team who has automated many FitNesse tests at the object level. I plan to get her to show me how these tests work, and see if what she did would work for our code.

I haven’t worked at a company this large since the early 90s, and I’ve never worked in a company, agile or otherwise, that had “architecture teams” and “frameworks”. I get it that our product needs to look and work consistently across the board, so it does make some sense to centralize the design of the UI and perhaps other parts of the system. However, if the architecture is done by one team, what does that mean for programmers on other teams? Is the fun part of their job taken away? Do they get a chance to think for themselves and improve the product?

It also seems to me that there needs to be a lot of communication amongst the entire development community of our company, that all programmers should have some input to the architecture process. I’ve found that communication between teams in general is a bit lacking. Each team is so focused on delivering its part of the product, they don’t always feel they have time to go talk to other teams. So, for example, perhaps nobody on my team has talked to the person who did a lot of FitNesse functional tests.

Does your company have an architecture team, and/or a framework that everyone works with? Please comment and share your experiences. As a tester, I’d like to know what I can do to help my team deliver the highest possible quality when we have to work with a framework designed by someone else.

There are a lot of creative, smart people on our team! Unfortunately I missed the discussion where these ideas come out, but we are trying some ways to organize such a big team. We will have “cells” or sub-teams, each one will take on a set of stories.

Today, each cell wrote high level acceptance tests together. Our cell started out doing BDD style tests on a FitNesse wiki. We used my desktop and the group in Weston VNC’d in and projected it on screen. But they couldn’t scroll up and down. And tests were taking too long to write. In our pomodoro retro, we decided to do high level test cases together as bullet points only, then have a pair flesh the tests out into BDD style and automate them.

We also decided to try Google docs. We kept using VNC and projecting the screen, but since the page is saved often, each person could also look at the doc on his own machine and scroll up and down. Both changes worked better. I had to split off for a long phone call with someone from an internal development team (that’s a whole ‘nother blog post), and when I came back they had finished the tests.

We are doing what I just learned is set-based engineering to decide whether we’d rather write our SWAT tests in C# and Visual Studio, or in FitNesse. We will do both this sprint and see which is better. My teammate Maykel started writing FitNesse fixtures that would allow using BDD tests to run SWAT from FitNesse, and I think he also had some way to make the tests run faster in FitNesse. One pair will automate the acceptance tests, while the other pair does a spike.

Thanks to the people who commented on yesterday’s post, I feel much better about having the testers get together occasionally. It’s true that we should have a testing community within our agile team, just as we have started one company-wide. I think we are on the right path with this team, despite it being larger than I would like.

I’ve been working in this large agile setting (28 Scrum teams, 2 week sprints, 4 production releases per year) about 6 weeks now. It has confirmed my past experiences that success depends on having good people, and letting them do their best work.

In this short time, my 5-person team (3 programmers, 2 testers) has implemented a CI and build process in Hudson, which build our code as well as the full product, runs our automated unit and FitNesse/SWAT tests, deploys to our sandboxes, and reports results. We’ve finished several stories. We’ve had one sprint review, which went well. We’ve shaved a lot of yaks, which is to be expected for a new team and new employees (especially yours truly. The combination of telecommuting and lack of Windows and network expertise isn’t good). We work in pairs and trios, and stay in constant communication via Skype, chat, webcam and shared desktops.

Something new to me is having to depend on other teams for certain tasks. If we need something added to the database, we have to put a request into Jira and wait, though the turnaround is fairly quick. In order to get data changes into our own environments, we sometimes have to do a db refresh process that can take hours. We even had to wait a few days for another team to provide the new error message we needed. Before we can get final approval from our PO on a story, we have to request that the official company-wide build be deployed on one of our environments. I was frustrated by this, but we seem to be adjusting our rythym so that we fill in the delays with productive work.

To give you an idea what my job is like now, here’s a typical day in my work life:

Before 8:00 AM Mountain time – Get on group chat with rest of team. I’m the last one on – 3 are on Eastern time, and 1 is just faster at getting to work than I am.

8:00 AM – Attend Scrum of Scrums by conference call. This consists of general news that affects all teams, such a build problems, and polling each of 28 teams for impediments. It’s usually over in 10 minutes, amazingly.

8:15 – Join Skype call and webcam videos with rest of team. Pair with fellow tester (Chris McMahon) to write automated GUI/functional tests in FitNesse/Skype. We use VNC to share desktops.

9:30 – Standup meeting. We have an app that brings up a photo of “The Usual Suspects” with our heads on random bodies. I look good with a tattoo.

9:45 -10:00 – Watch the developers work on some unit tests, using LiveMeeting to see their desktop.

10:00 – 11:30 – Catch up on emails, maintenance tasks, and the like, and eat lunch.

11:30 – we’re all back on Skype, chat, webcam and desktop sharing.

11:30 – 12:00 One of the developers helps me with an issue I’m having with my local test environment, using VNC to take control of my desktop.

12:00 – 12:30 We discuss our acceptance tests with the PO. The other teams he works with put acceptance tests in spreadsheets and attach them to the story “card” in Jira. We are writing the acceptance tests together in sprint planning, in BDD style, directly onto the FitNesse wiki pages where we’ll also write the automated tests. Then Chris and I automate the tests. The PO is fine with our approach, he can easily understand the tests, and has found some test cases that we missed. However he’s concerned that we’re deviating from the “standard”. So far, nobody has objected to our writing tests in the FitNesse wiki, and it works so well for us, we’re going to continue.

12:30 – 1:30 Pair with Chris again to update the narrative acceptance tests for the stories based on the PO’s input, and write additional automated tests.

1:30 – 3:00  The database change needed for the story the developers finished yesterday is ready. Chris and I update our local databases, deploy the new code, run the FitNesse/SWAT tests, and do exploratory testing of the new functionality. Whenever we run across an issue, we show it to the developers using VNC desktop sharing.

3:00 – 5:00 Everyone else is done at 3:00, so I’m on my own. I update the wiki with notes about how to use the system that allows us to add test users. Then I do some additional exploratory testing on the story we worked on earlier. Next, I look into a problem we were having in a SWAT test. I had emailed the internal user group about it, and had some responses with suggestions to try. I’m able to get this working, and check the test in. I make sure our task board is up to date.

As time goes by, I spend less time on things like getting my environment working or solving network problems, and more time doing productive work. I’m also starting an internal company testing community, so we can all share ideas. My teammates and I are blogging internally about how we are writing and designing tests, and how we’ve implemented our CI and build process. We hope the ideas that work for us will spread to other teams, and we’re also looking at what other teams to and adopting their good ideas.

So far, so good!

I’ve worked on four agile teams in the past nine years. Three were highly successful, especially the one I was on for the past 5.5 years. One never really made the transition, though we had a successful agile project. One of the successful ones involved a team of about 30 developers, and that’s the largest project I’ve been on up to now.

I could have happily worked with my previous team until I retire, but you know me, I cannot resist new opportunities. So now I’m on a small Scrum team – that is one of 28 Scrum teams at my new company! Yes, this company actually manages to have 28 teams, including some which provide infrastructure such as frameworks and deployment, working together to release a high-quality commercial software product four times a year.

Yeah, back in the day, XP was for little, co-located teams. “You can’t do shrink-wrapped software with XP” was just one of the many axioms I heard.

Now I work in an agile organization with about 140 developers, some (like me) in remote locations, delivering software both for Windows client and for a hosted solution. (I don’t know everything about it yet so sorry to sound vague!) If you had told me this five or ten years ago, I wouldn’t have been sure it could work.

Since my new 5-person team is a self-organizing Scrum team, we picked our own ScrumMaster, which ended up being me. I love being an uncertified ScrumMaster, and plan to delegate a lot, because I’m also committed to remaining a hands-on tester. I attended my first Scrum of Scrums (some attendees called it the “SoS” which I loved) on Friday. Each of the 28 ScrumMasters (some in the room, some on the phone) was called on to say if they had any impediments. This happened to be a release day, so I thought there might be lots of impediments, but most teams didn’t have any. Those that did got help right away. The meeting was over in 10 or 15 minutes.

Granted, not every team is high-performing, and some practices I’m used to are missing altogether. But think about it, a software development organization that large, which only implemented Scrum four years ago, able to coordinate and release critical value frequently, at a sustainable pace. That’s agile in anyone’s book.

Follow this space for my adventures as I enter a whole new world!