Many of the teams do a great job of automating their tests, and a few teams have their own continuous integration (CI) that runs multiple times a day. Other teams have automated tests, but must wait for the company-wide build to run, which is usually only once per day and sometimes not that often. They don’t get feedback fast enough, and that slows them down – a lot.

I’ve been asking why some teams struggle to automate tests, and why some teams don’t have their own CI. I’ve also asked my own team why most of our functional tests are automated through the GUI, rather than behind the GUI. We use FitNesse, but mainly to drive GUI tests with SWAT.

The answer I get a lot is “It’s the framework”. We have three architecture teams, and there is a framework (maybe there is more than one, I don’t fully understand it yet) that apparently makes it hard to test at the page or object layer. In addition, some teams’ code has tentacles into legacy code written in Delphi whose tests are written in WinRunner. Also, the current framework appears to not be fully supported with automated unit tests, it was developed a few years back. Programmers new to the framework seem to have a hard time understanding how to work with it. The framework clearly causes a lot of frustration. A new framework is under development, but it won’t be available for some time.

In order to understand the existing framework and find out whether the new one will facilitate testing behind the GUI, I was fortunate to get to meet with the leader of the architecture side of the development group. He gave me an overview of the new framework’s current design (which may change), aimed at making our product easily configurable by customers, using web services. It looks like this design allows testing under the GUI level.

Additionally, I learned that it is possible to test under the GUI in the current framework, and I met someone on another team who has automated many FitNesse tests at the object level. I plan to get her to show me how these tests work, and see if what she did would work for our code.

I haven’t worked at a company this large since the early 90s, and I’ve never worked in a company, agile or otherwise, that had “architecture teams” and “frameworks”. I get it that our product needs to look and work consistently across the board, so it does make some sense to centralize the design of the UI and perhaps other parts of the system. However, if the architecture is done by one team, what does that mean for programmers on other teams? Is the fun part of their job taken away? Do they get a chance to think for themselves and improve the product?

It also seems to me that there needs to be a lot of communication amongst the entire development community of our company, that all programmers should have some input to the architecture process. I’ve found that communication between teams in general is a bit lacking. Each team is so focused on delivering its part of the product, they don’t always feel they have time to go talk to other teams. So, for example, perhaps nobody on my team has talked to the person who did a lot of FitNesse functional tests.

Does your company have an architecture team, and/or a framework that everyone works with? Please comment and share your experiences. As a tester, I’d like to know what I can do to help my team deliver the highest possible quality when we have to work with a framework designed by someone else.

Today, each cell wrote high level acceptance tests together. Our cell started out doing BDD style tests on a FitNesse wiki. We used my desktop and the group in Weston VNC’d in and projected it on screen. But they couldn’t scroll up and down. And tests were taking too long to write. In our pomodoro retro, we decided to do high level test cases together as bullet points only, then have a pair flesh the tests out into BDD style and automate them.

We also decided to try Google docs. We kept using VNC and projecting the screen, but since the page is saved often, each person could also look at the doc on his own machine and scroll up and down. Both changes worked better. I had to split off for a long phone call with someone from an internal development team (that’s a whole ‘nother blog post), and when I came back they had finished the tests.

We are doing what I just learned is set-based engineering to decide whether we’d rather write our SWAT tests in C# and Visual Studio, or in FitNesse. We will do both this sprint and see which is better. My teammate Maykel started writing FitNesse fixtures that would allow using BDD tests to run SWAT from FitNesse, and I think he also had some way to make the tests run faster in FitNesse. One pair will automate the acceptance tests, while the other pair does a spike.

Thanks to the people who commented on yesterday’s post, I feel much better about having the testers get together occasionally. It’s true that we should have a testing community within our agile team, just as we have started one company-wide. I think we are on the right path with this team, despite it being larger than I would like.

I need to investigate more what they mean by this, I might be misinterpreting it. My new team is the combination of three teams, totalling 17 people (not counting the business folks). I think that’s unwieldy. Today, an experienced tester on my new team, who has worked on an agile team for some period of time (I’m not sure how long), proposed that the “test team” get together and discuss practices and strategy. I about blew a gasket, but that didn’t keep the meeting from happening (I’m always on phone and video with my team, so I can’t hep being in the meetings.)

The content of the meeting was reasonable enough. One topic was patterns of tests that we do, for example, for a numeric field you would always try certain test cases such as alpha and special characters, null, etc. Standard stuff.

The next topic was about automating tests on a page and end-to-end level, in addition to at the story level. This bugged me. If you tie tests to a page in the UI, and later stuff gets moved around so that what used to be on that one page is now on different pages, your tests will break. As far as automating end-to-end tests. we definitely have to be careful since the ROI can be bad on automating those.

My teammate Chris McMahon explained how our team up to now has been trying to group tests at the domain level. One FitNesse test contains all tests related to a rating scale, for example, until there are about 200 test steps and then we split it up into some reasonable sub-grouping. We’re testing at a feature level, no matter where we are in the UI.

The next topic was localization. Our app accommodates data in three languages. Someone proposed that we design our tests so that they could be run for any language. This sounds good to me, but how hard is it to do, and is it worth it? Is the localization already tested by the team that does the framework for it? I suggested we definitely needed programmer agreement for this topic, and everyone agreed.

I then brought up topics I’ve been wanting to raise, the non-functional testing. Our company has teams to do performance, scalability, reliability and security testing, but isn’t our team still responsible to make sure those happen in a timely manner? This discussion proved helpful for me, as I learned about the teams that do this testing, and how we can become involved and do a lot of it ourselves. I’m glad to get these activities out in the open and on our minds so they won’t be overlooked.

In the end I needn’t have been so spooked by the idea of testers meeting together. As it turned out, the programmers got into a design discussion with the database guy, and we were not involved in that (welll, I didn’t know about it until it was underway), so it was an opportunity for the testers to talk. But I don’t want that to happen too often.

I’m looking forward to starting work on stories. I think our test design will evolve as we write new code. We will pair on everything, so I hope that means the programmers and testers will remain engaged with each other.

In this short time, my 5-person team (3 programmers, 2 testers) has implemented a CI and build process in Hudson, which build our code as well as the full product, runs our automated unit and FitNesse/SWAT tests, deploys to our sandboxes, and reports results. We’ve finished several stories. We’ve had one sprint review, which went well. We’ve shaved a lot of yaks, which is to be expected for a new team and new employees (especially yours truly. The combination of telecommuting and lack of Windows and network expertise isn’t good). We work in pairs and trios, and stay in constant communication via Skype, chat, webcam and shared desktops.

Something new to me is having to depend on other teams for certain tasks. If we need something added to the database, we have to put a request into Jira and wait, though the turnaround is fairly quick. In order to get data changes into our own environments, we sometimes have to do a db refresh process that can take hours. We even had to wait a few days for another team to provide the new error message we needed. Before we can get final approval from our PO on a story, we have to request that the official company-wide build be deployed on one of our environments. I was frustrated by this, but we seem to be adjusting our rythym so that we fill in the delays with productive work.

To give you an idea what my job is like now, here’s a typical day in my work life:

Before 8:00 AM Mountain time – Get on group chat with rest of team. I’m the last one on – 3 are on Eastern time, and 1 is just faster at getting to work than I am.

8:00 AM – Attend Scrum of Scrums by conference call. This consists of general news that affects all teams, such a build problems, and polling each of 28 teams for impediments. It’s usually over in 10 minutes, amazingly.

8:15 – Join Skype call and webcam videos with rest of team. Pair with fellow tester (Chris McMahon) to write automated GUI/functional tests in FitNesse/Skype. We use VNC to share desktops.

9:30 – Standup meeting. We have an app that brings up a photo of “The Usual Suspects” with our heads on random bodies. I look good with a tattoo.

9:45 -10:00 – Watch the developers work on some unit tests, using LiveMeeting to see their desktop.

10:00 – 11:30 – Catch up on emails, maintenance tasks, and the like, and eat lunch.

11:30 – we’re all back on Skype, chat, webcam and desktop sharing.

11:30 – 12:00 One of the developers helps me with an issue I’m having with my local test environment, using VNC to take control of my desktop.

12:00 – 12:30 We discuss our acceptance tests with the PO. The other teams he works with put acceptance tests in spreadsheets and attach them to the story “card” in Jira. We are writing the acceptance tests together in sprint planning, in BDD style, directly onto the FitNesse wiki pages where we’ll also write the automated tests. Then Chris and I automate the tests. The PO is fine with our approach, he can easily understand the tests, and has found some test cases that we missed. However he’s concerned that we’re deviating from the “standard”. So far, nobody has objected to our writing tests in the FitNesse wiki, and it works so well for us, we’re going to continue.

12:30 – 1:30 Pair with Chris again to update the narrative acceptance tests for the stories based on the PO’s input, and write additional automated tests.

1:30 – 3:00  The database change needed for the story the developers finished yesterday is ready. Chris and I update our local databases, deploy the new code, run the FitNesse/SWAT tests, and do exploratory testing of the new functionality. Whenever we run across an issue, we show it to the developers using VNC desktop sharing.

3:00 – 5:00 Everyone else is done at 3:00, so I’m on my own. I update the wiki with notes about how to use the system that allows us to add test users. Then I do some additional exploratory testing on the story we worked on earlier. Next, I look into a problem we were having in a SWAT test. I had emailed the internal user group about it, and had some responses with suggestions to try. I’m able to get this working, and check the test in. I make sure our task board is up to date.

As time goes by, I spend less time on things like getting my environment working or solving network problems, and more time doing productive work. I’m also starting an internal company testing community, so we can all share ideas. My teammates and I are blogging internally about how we are writing and designing tests, and how we’ve implemented our CI and build process. We hope the ideas that work for us will spread to other teams, and we’re also looking at what other teams to and adopting their good ideas.

So far, so good!