Agile Testing with Lisa Crispin

I had the opportunity to talk with Gil Zilberfeld, product owner with Typemock, a software company whose products are designed to help developers with their unit testing. I don’t often get to talk to product owners, and I was interested in his views on testing and quality. (Disclaimer: I have no experience with using Typemock’s products, and this is not a commentary on those products!)

An Agile Approach

Gil works with a team of three developers, who also provide technical support to customers. They practice agile development, working in small increments and short iterations, releasing new versions of their products a few times per year. They use agile practices such as pair programming and continuous integration.

Gil’s team doesn’t have any testers, but they clearly do plenty of testing, both unit and integration testing . Gil explained to me that his team practices “dog fooding”: they use their own product to help with their own unit testing.  They have thousands of unit tests running in their continuous integration, providing quick feedback after every check-in. I thought it was interesting that, although their team is co-located, they use an online board product, Trello. Gil said the sticky notes kept falling down.

Developing What Customers Want

I asked Gil how his team comes up with requirements for things such as user interfaces and APIs. He said that rather than compose formal user stories, they “throw ideas in the air”, sketch out what they want on whiteboards, break features down and code them. They don’t make changes to requirements during the iteration, but they can continue to tweak the features in subsequent iterations. They try to innovate and come up with new features that can help both advanced and new users. Once they release a new feature, they use feedback from customers to enhance it. They listen to requests and fix problems quickly.

Using their own product allows the developers to evaluate the user experience and performance aspects, rather than doing formal usability or performance testing. Since the developers rotate the technical support duties, they each get time working directly with end users who are having issues with the software. This sounds to me like a great way to get a feel for quality from the customers’ perspective. I wish my own team had this kind of direct contact with users.

Value to Customers

Gil noted that when Typemock’s products were new, the early adopters were more flexible – not everything had to be perfect. Now, they’re producing enterprise software, the customers have changed. Developers with different experience levels have different needs. Inexperienced programmers may value an easy learning curve  over sophisticated features. Even though they use their own product, Gil’s team doesn’t always know what the customers will like. They use beta testing as needed, choosing the types of customers that can provide the most useful feedback.

Gil told me a great story about his previous experience working for a company that produced large medical devices. His team didn’t have direct contact with customers. His first visit to a real customer was eye-opening. The customer had many large devices, and very little room. Gil could see that requirements he had thought weren’t very important were actually critical because of the space limitations. For example, computers needed to be inside the machine, with touch screens, to save space. This is a good lesson in why we need to understand value from the customer’s perspective.

Continually Improving

Since Gil’s team doesn’t have testers and does their own testing, I asked what their main pain points were with respect to testing and quality. He said they want to make some improvements to their continuous integration process, and reduce some technical debt they have incurred there. Next, they would like to address velocity issues. They work at a good velocity, but management always wants more output. Gil said this is a question of matching expectations. They have to balance developing new features with meeting support standards and fixing bugs.

Since I work on a small team that works on a now-mature product, I was interested to hear about experiences of a similar team. Though they’re in a different domain, it sounds to me like they have a similar commitment to delivering the best possible quality software product that they can. My experience is that there are more and more teams like Gil’s that think about many different aspects of quality. They go beyond functional testing to think about different quality aspects the customers value, and try innovative approaches to delivering those.

I have been doing some writing in the past six weeks, in addition to rehabilitating my broken-and-now-healed ankle. But it’s all been published elsewhere. Here are some links:

Software Quality Connection, “How to Improve Communication between QA and Development“

SearchSoftwareQuality.com, “Agile development: The whole-team approach”

Also on SearchSoftwareQuality.com, “The whole-team approach to Agile development: Examples and benefits“

I’ve got a bunch of tips and Ask the Expert Q&A on SearchSoftwareQuality.com, please check them out! I’d appreciate feedback.

I’m presenting in a virtual multi-media seminar sponsored by SearchSoftwareQuality on Dec.14: Agile in Action: Extending Agile Approaches in Testing, Development and Application Lifecycle Management. I’ll be talking about challenges faced by agile teams that are not co-located. This event is free, though you have to register. David West and Nari Kannan are also presenting.

Today I was lucky to hear part of the Thomas Jefferson Hour as I drove down to the ranch to work the donkeys. Thomas Jefferson, channeled by the actor and scholar Clay Jenkinson, quoted from his first inaugural address: “Every difference of opinion is not a difference of principle”. Jefferson believed that our elected officials should respond to moments of tension (and as he says, there was just as much rancor in political debates back in his day as now) with politeness, good humor and respectfulness. He recommended that our current elected officials, if they can’t find common ground, should adopt “artificial good humor” and assume the best about their opponents.

TJ and I might both have our head in the clouds, but I’m a big believer in civility. I’m not recommending that we all walk around on eggshells and worry about offending people. But I do think we need to all work hard to make sure that everyone on our software team feels safe to express opinions, and conversely, is willing to accept the team consensus or majority rule, and work to make whatever approach is taken successful.

This is something I want to take to heart for myself. Here’s an example. I’m trying to explain an issue to the programmer who is on production support this sprint. I’ve analyzed the problem and think I know how it should be fixed. He seems not to understand, gets impatient with me, throws up his hands and tells me I am mistaken. I could feel mad and hurt by this, or I can remember that being the production support monkey is frustrating and stressful. I’ve given him the information I wanted to convey, and I can leave him to process it. He’s smart and motivated to take care of our customers, so I know he will find a good solution. He shouldn’t be rude to me, but if I can avoid escalating the tension, we will have a good outcome.

Indeed, later the programmer comes over and apologizes, says now he understands what I was trying to say, and appreciates the research I did into the issue. We talk about the fix and how to test it.

I’m lucky to work on a team that, though there is lots of joshing and joking, is composed of people with a deep commitment to providing a quality product, and open minds willing to consider anyone’s ideas. I’ve been on teams that didn’t provide this atmosphere of respect and civility, and I must confess, I hightailed it away from those toxic environments.

The “agile” movement was built on principles and values. “Every difference of opinion is not a difference in principle”. Give your team time and space to agree on principles and values. Going forward from there, each team member will feel free to float their ideas and opinions, and have healthy discussions. That’s what makes a project successful.

News!

Janet Gregory and I have an article, and are interviewed in, this month’s Software Test and Performance Magazine, devoted to “Women of Influence in Software Testing”. Please check it out, it’s a great issue with lots of meaty articles.

I have a sidebar in Dawn Cannan‘s terrific “Be the Worst” article in the inaugural issue of Agile Record.

Matt Davey wrote a wonderful thumbnail summary of our book in his review of it, we are grateful.

I work for a mid-sized software company with around 27 development teams, mostly Scrum but also a few doing Lean and Kanban. We have a large and diverse product of HR-related software, I think there are around 6 million LOC, on three different platforms, including an ASP product and a .NET product. I’m fairly new, I started at the beginning of June, so I’m still learning about how everything gets done. We release four times a year and the software has won many prestigious awards. I’ve been impressed with this large-scale implementation of agile development. Just the speed of the daily Scrum of Scrums is amazing.

Many of the teams do a great job of automating their tests, and a few teams have their own continuous integration (CI) that runs multiple times a day. Other teams have automated tests, but must wait for the company-wide build to run, which is usually only once per day and sometimes not that often. They don’t get feedback fast enough, and that slows them down – a lot.

I’ve been asking why some teams struggle to automate tests, and why some teams don’t have their own CI. I’ve also asked my own team why most of our functional tests are automated through the GUI, rather than behind the GUI. We use FitNesse, but mainly to drive GUI tests with SWAT.

The answer I get a lot is “It’s the framework”. We have three architecture teams, and there is a framework (maybe there is more than one, I don’t fully understand it yet) that apparently makes it hard to test at the page or object layer. In addition, some teams’ code has tentacles into legacy code written in Delphi whose tests are written in WinRunner. Also, the current framework appears to not be fully supported with automated unit tests, it was developed a few years back. Programmers new to the framework seem to have a hard time understanding how to work with it. The framework clearly causes a lot of frustration. A new framework is under development, but it won’t be available for some time.

In order to understand the existing framework and find out whether the new one will facilitate testing behind the GUI, I was fortunate to get to meet with the leader of the architecture side of the development group. He gave me an overview of the new framework’s current design (which may change), aimed at making our product easily configurable by customers, using web services. It looks like this design allows testing under the GUI level.

Additionally, I learned that it is possible to test under the GUI in the current framework, and I met someone on another team who has automated many FitNesse tests at the object level. I plan to get her to show me how these tests work, and see if what she did would work for our code.

I haven’t worked at a company this large since the early 90s, and I’ve never worked in a company, agile or otherwise, that had “architecture teams” and “frameworks”. I get it that our product needs to look and work consistently across the board, so it does make some sense to centralize the design of the UI and perhaps other parts of the system. However, if the architecture is done by one team, what does that mean for programmers on other teams? Is the fun part of their job taken away? Do they get a chance to think for themselves and improve the product?

It also seems to me that there needs to be a lot of communication amongst the entire development community of our company, that all programmers should have some input to the architecture process. I’ve found that communication between teams in general is a bit lacking. Each team is so focused on delivering its part of the product, they don’t always feel they have time to go talk to other teams. So, for example, perhaps nobody on my team has talked to the person who did a lot of FitNesse functional tests.

Does your company have an architecture team, and/or a framework that everyone works with? Please comment and share your experiences. As a tester, I’d like to know what I can do to help my team deliver the highest possible quality when we have to work with a framework designed by someone else.

This month’s issue Dressage Today had some articles about the two-time Olympian dressage rider and trainer, Steffen Peters. He talked about the role of discipline in turning dreams into reality. For him, this means keeping himself mentally and physically fit through a thoughtfully planned exercise and diet routine, as well as understanding the capabilities of each horse and how it can learn best.

My own dressage trainer tells me all the time, “It takes a million repetitions to change a bad habit.” That’s only a slight exaggeration in my case. I remind myself (and she reminds me) about 50 times a ride to correct each bad habit, for example, to put my hands closer together and keep them centered over the horse’s withers. Some of my bad habits are 50 years old, so they’re hard to fix, but I’m slowly getting better.

Discipline is a critical component of delivering high-quality software, too. We need to thoughtfully plan to use the right practices for us. We need to remind ourselves every day, via big visible charts, information radiators, retrospectives, any means that works to learn new good habits and reinforce the things we do well.

During a recent estimating meeting, our product owner (who normally knows better) asked, “About 6 weeks ago you estimated this XYZ story at 5 points. Is there any cheap, hacky way to do it for one point?” One of the developers responded, “Possibly, but we are not going to do a cheap, hacky solution”. It’s tempting to cut corners to meet a business need, but hacking in a quick fix that increases our technical debt down the road isn’t the way to do it.

Another example where discipline and good habits came into play occurred in this morning’s Scrum. A couple of the customers brought up a significant piece functionality that they had accidentally omitted from a story we’re already working on. It was tempting to say “OK, we’ll add that in too.” But it was too big a change to put into the story at this late date. We discussed possible workarounds, and one of the customers came up with one that would do the job. They will write a new story, and if we can, we’ll pull it in this sprint. If not, they’ll get it two weeks later.

Sometimes we do make compromises about quality, and implement a solution that is less than ideal. There are always business trade-offs to consider. But we don’t make these lightly. And even after more than five years of agile development, we spend time on a serious retrospective every iteration, and work hard to keep improving the way we work and the quality of our code. Some of our bad habits have been hard to fix, but we keep working at it. Our business continues to grow, which to me is the best feedback about our disciplined approach to keeping up good habits and eliminating bad habits.