Agile Testing with Lisa Crispin

My friend and excellent agile coach Michele Sliger, co-author of The Software Project Manager’s Bridge to Agility ,  recently emailed me this question that she hears a lot: “Where should I start looking to learn more about test automation and tooling?”

Naturally, I replied with a typically long-winded answer. She must have liked it though, because she suggested I cut and paste my reply into a blog post. Here it is!

The key thing with a newbie agile team is to not say “move testers on all teams into automated testing” but “help all teams learn to take whole-team responsibility for quality and testing, and learn how programmers and testers collaborate to automate tests at all levels”. Automated test code is code, and generally, it’s much quicker and better in the long run for programmers to code the automation fixtures. Testers know what to test, and when programmers do the coding, testers have lots of time to help customers come up with examples of desired behavior to turn into tests. Then they can pair with the programmers to automate regression testing and any other types of tests (performance, load etc). Then they will still have time to do the all-important manual exploratory testing.

If they read our book Agile Testing, they will learn that they have to be patient and invest a lot of time in learning and experimenting with different test frameworks. And they should use Mike Cohn’s test automation pyramid concept to see where they’ll get the best ROI in automation. Generally, they should always start by learning how to do TDD and get traction on unit level tests. Then they can move on to API or service level tests, then GUI. But that’s not a rule, each team has to figure out what’s best for them.

I highly recommend Gojko Adzic’s Specification by Example book, and also his blog, gojko.net. Markus Gaertner has a great ATDD book coming out but it’s not published yet.

There are SO many test frameworks, drivers, tools, it’s overwhelming. Recently we had to find a way to automate GUI regression tests for our new code that uses Dojo. Our existing GUI tools aren’t able to interpret the Dojo JS properly. It was truly a team effort, though not a smooth one. Our sys admin had earlier played around with Selenium, and we thought Webdriver might work. The sys admin did a spike and proved it did work, then he and our senior architect spiked a couple of different frameworks – one homegrown, one using Geb. They demo’ed it and got everyone’s opinions. For now we are going with the Geb framework, but down the road we might decide it’s not quite right and try something else.

Anyway, that’s a long-winded answer. One good place to start getting an idea about functional test tools is the Agile Alliance Functional Test Tools group’s spreadsheet comparing various tools, at http://bit.ly/AgileTestTools. When the teams are ready to think about functional testing, they might take a look and get an idea of all the choices. It’s a good idea to stop and plan a test automation strategy.

It is really, really hard to get started with test automation. There’s a period where it’s just extra work and no reward. But eventually they’ll cross over that Hump of Pain and start reaping the benefits, eating away at their technical debt!

I had the opportunity to talk with Gil Zilberfeld, product owner with Typemock, a software company whose products are designed to help developers with their unit testing. I don’t often get to talk to product owners, and I was interested in his views on testing and quality. (Disclaimer: I have no experience with using Typemock’s products, and this is not a commentary on those products!)

An Agile Approach

Gil works with a team of three developers, who also provide technical support to customers. They practice agile development, working in small increments and short iterations, releasing new versions of their products a few times per year. They use agile practices such as pair programming and continuous integration.

Gil’s team doesn’t have any testers, but they clearly do plenty of testing, both unit and integration testing . Gil explained to me that his team practices “dog fooding”: they use their own product to help with their own unit testing.  They have thousands of unit tests running in their continuous integration, providing quick feedback after every check-in. I thought it was interesting that, although their team is co-located, they use an online board product, Trello. Gil said the sticky notes kept falling down.

Developing What Customers Want

I asked Gil how his team comes up with requirements for things such as user interfaces and APIs. He said that rather than compose formal user stories, they “throw ideas in the air”, sketch out what they want on whiteboards, break features down and code them. They don’t make changes to requirements during the iteration, but they can continue to tweak the features in subsequent iterations. They try to innovate and come up with new features that can help both advanced and new users. Once they release a new feature, they use feedback from customers to enhance it. They listen to requests and fix problems quickly.

Using their own product allows the developers to evaluate the user experience and performance aspects, rather than doing formal usability or performance testing. Since the developers rotate the technical support duties, they each get time working directly with end users who are having issues with the software. This sounds to me like a great way to get a feel for quality from the customers’ perspective. I wish my own team had this kind of direct contact with users.

Value to Customers

Gil noted that when Typemock’s products were new, the early adopters were more flexible – not everything had to be perfect. Now, they’re producing enterprise software, the customers have changed. Developers with different experience levels have different needs. Inexperienced programmers may value an easy learning curve  over sophisticated features. Even though they use their own product, Gil’s team doesn’t always know what the customers will like. They use beta testing as needed, choosing the types of customers that can provide the most useful feedback.

Gil told me a great story about his previous experience working for a company that produced large medical devices. His team didn’t have direct contact with customers. His first visit to a real customer was eye-opening. The customer had many large devices, and very little room. Gil could see that requirements he had thought weren’t very important were actually critical because of the space limitations. For example, computers needed to be inside the machine, with touch screens, to save space. This is a good lesson in why we need to understand value from the customer’s perspective.

Continually Improving

Since Gil’s team doesn’t have testers and does their own testing, I asked what their main pain points were with respect to testing and quality. He said they want to make some improvements to their continuous integration process, and reduce some technical debt they have incurred there. Next, they would like to address velocity issues. They work at a good velocity, but management always wants more output. Gil said this is a question of matching expectations. They have to balance developing new features with meeting support standards and fixing bugs.

Since I work on a small team that works on a now-mature product, I was interested to hear about experiences of a similar team. Though they’re in a different domain, it sounds to me like they have a similar commitment to delivering the best possible quality software product that they can. My experience is that there are more and more teams like Gil’s that think about many different aspects of quality. They go beyond functional testing to think about different quality aspects the customers value, and try innovative approaches to delivering those.

Group working on experiments to build trust

On December 13, SQuAD (Software Quality Association of Denver) members got together for a workshop to generate ideas that might help distributed teams succeed with agile and testing. We started by identifying some areas that pose particular challenges with distributed teams, such as coordinating, visualizing, learning, and building trust. Then, each table group chose an area of challenge, and brainstormed experiments to help overcome those challenges. We finished up by having each group share their top experiment with all the participants. Some of the results were quite imaginative, and we all got some good ideas to try.

One group tackled what they termed “The 24-Hour Question”. When you have a team whose time zones are half a day apart, when one location has a question for the other, they often have to wait a day for the answer. They suggested aligning philosophies, coming to agreement on testing practices, coding standards and the like, along with finding better ways to communicate.

The 24-Hour Question

Another group’s experiment to improve communication proposed guided, controlled feedback. Establish a baseline, identify primary points of contact, and eliminate unnecessary parties. Then engage, manage expectations, and get feedback. Try new ways of communicating, and try to measure results and use the feedback to see if the experiment helped. Since I couldn’t take notes, I’m afraid I’ve forgotten the details, but the gist is to actually try to measure results of your experiments.

Some experiments included getting everyone on a team together for project kickoff, or getting at least some members from all locations together once a quarter. One participant has been doing this with his own team, and management had found the investment in travel cost worthwhile. Many experiments involved making use of technology, including video conferencing, instant messaging including IRC style tools such as Campfire, desktop sharing, projecting blogs and other online resources, and even virtual planning rooms. Some ideas were simple: share photos among remote locations, and email or IM jokes to each other.

A group discussing experiments to build trust proposed a kickoff meeting, remote pairing, management buy-in, a survey, and team-building activities. One of their experiments was to have everyone on the team work remotely.  Another was to have the remote team lead the meetings. Retrospectives were also recommended, and I think that’s one of the best ways to evaluate how well an experiment worked and think of new ones to try.

A Festive Experiment

My own take-home aha was the idea that we need to first set a baseline – how is our distributed team working now? This might involve surveying all team members to see what works or doesn’t work for them. Then try one small experiment for one iteration, and use the retrospective to talk about whether it helped with a particular issue. Keep finding ways to measure progress. As with any problems faced by agile teams, we can’t fix everything at once. Take baby steps, but get feedback all along the way.

Please see more photos of our fun workshop.

My new article “Story Mapping the Wrong Way”, a tale of how I messed up our team’s first attempt at story mapping but we learned stuff anyway, is on Techwell and Stickyminds. How fascinating! :->

Gojko Adzic was voted “Most Influential Agile Testing Professional Person” by his peers, in an award program sponsored by the Agile Testing Days conference. I was asked to give the honorific speech before the actual award ceremony. This is no doubt the highlight of my professional career – I got to give a speech to the conference participants ON A HORSE WEARING A COSTUME. It just won’t get any better than this! Here is the video – warning, it is long, I don’t have time to edit it. If it’s possible to fast forward to the end, do so and watch Gojko getting knighted by the Lord Mayor of Potsdam!

Someone on the agile-testing Yahoogroup mailing list posted a link to a blog post in which he proceeded to misuse, maul and maim the Agile Testing Quadrants. There is no way to put comments on that blog post to try to refute his claim that the quadrants are somehow a waterfall process. Since other people might misunderstand the purpose of the quadrants, I’d like to put a quick explanation here.

Agile Testing Quadrants

You might want to start with Brian Marick‘s original posts on his agile testing matrix, which we called the Quadrants and (with his permission) made the heart of our Agile Testing book.

The quadrant numbering system does NOT imply any order. You don’t work through the quadrants from 1 to 4, in a waterfall style. It’s just an arbitrary numbering so that, in our book and when we are talking about the quadrants, we can say “Q1″ instead of “technology-facing tests that support the team”. 

Most projects would start with Q2 tests, because those are where you get the examples that turn into specifications and tests that drive coding, along with prototypes and the like. However, I have worked on projects where we started out with performance testing (which is in Q4) on a spike of the architecture, because that was the most important criterion for the feature. If your customers are uncertain about their requirements, you might even do a spike and start with exploratory testing (Q3).

Q3 and Q4 testing pretty much require that some code be written and deployable, but most teams iterate through the quadrants rapidly, working in small increments. Write a test for some small chunk of a feature, write the code, once the test is passing, perhaps automate more tests for it, do exploratory testing on it, do security or load testing on it, whatever, then add the next small chunk and go through the whole process again.

The quadrants are merely a taxonomy to help teams plan their testing and make sure they have all the resources they need to accomplish it. There are no hard and fast rules about what goes in what quadrant. Think through them as you do your release, theme, and iteration planning, so your whole team starts out by thinking about testing first.

Michael Huetterman adds “Outside-in, barrier-free, collaborative” to the middle of the quadrants, see his Agile Record article or his excellent book Agile ALM

Visit my Presentations page for some slide decks that contain more information on the quadrants, or check out our book. I’m always happy to talk about the quadrants, just send me an email!

In my latest Techwell blog post, I tell the tale of our most recent estimating meeting, where the product owner brought us a user story that was written as a technical implementation. We had to get him to back up and tell us the purpose of the story, so that we could determine the appropriate technical implementation.

I learned about Adam Weisbart’s retrospective fortune cookies via Twitter. He kindly sent me a box of them for our team to try.

Vince and Lisa choose their cookies

We’ve been doing sprint retrospectives every two weeks for eight years, so it’s always good to get out of our rut and try something new! We each chose a cookie, and took turns reading the “fortune” inside, which was a thought-provoking question. The first question was “What could the ScrumMaster do to be more effective?” This discussion led to an idea for a new Big Visible Chart – a wall on which the SM would show the stories she and the Product Owner are preparing for upcoming sprints. We also decided to try going over requirements for each user story with not only the product owner, but the primary stakeholder for each story, which the SM will ident

Lori looks excited about her fortune

ify.

All of the “fortune” questions provoked good discussions, and ones we wouldn’t have had otherwise (we had already done our standard retrospective before digging into the fortune cookies.) For example, “How would you improve our sprint review?” Ours could certainly use improvement, but we never talk about it. I think our next sprint review will be better!

One of the questions puzzled us, “Were our Artifacts helpful for this sprint? Could we improve them? How?” We weren’t sure what “Artifacts” referred to. But the question led us into thinking of a better way to note requirements changed after coding begins, and ensure the developers are informed of all changes.

A cookie for our remote team member

The fortune cookie concept posed a challenge for Nanda, our teammate in India. We had to open his cookie and read it for him. While we were fighting over who should get to eat his cookie, the cookie got dropped and shattered on the carpet. Having fun as well as thinking of experiments to improve made the retrospective fortune cookies a big success!

Whoops!

(Thanks to my teammate Mike Thomas for taking some of these action photos with our team camera).

Last May, we were asked to enlarge our small team by hiring another tester and another programmer. We’ve made a few hiring decisions in the past which turned out to be mistakes, so we decided to enhance our interview process. After a lot of brainstorming and research, we came up with some ideas to try. Today we got to use our new process for the first time when we interviewed a tester candidate.

We’ve always done group interviews, where all team members are in the room, and we all ask questions, focusing on open-ended, behavioral style questions (see my article about hiring agile testers for examples). We do this for about 45 minutes, also allowing time for the candidate to ask us questions. Then, we hook up a laptop to the projector, navigate to an interesting page in our UI, and ask the candidate to do manual exploratory testing on it, explaining to us what she’s doing and why as she tests. We also asked the candidate to come up with a simple SQL query that involves joining two tables, and for simple Unix commands (“How do you navigate to a directory, how do you see what files are in the directory).

When we discussed what’s important to us in a tester candidate, some criteria came out that we couldn’t evaluate well with our existing process. We need a tester who has some programming experience (though she doesn’t have to be a programmer), who can maintain her own test environments, who knows how to work with customers in an agile setting to flesh a user story out with examples and requirements. Here are the activities we added with those goals in mind.

We had someone play the customer role, describe a user story in the “As a… I want… so that…” format, and asked the candidate to talk with the customer and get requirements for the story.  The candidate today asked some great questions, covering not only functional requirements, but other considerations such as response time and load. In fact, I was rather awestruck at how quickly the candidate fired off questions and thought of many different aspects of quality. We stopped him after about 5 minutes because we could tell he knew how to ask good questions.

Then we asked the candidate what tests he thought he might automate for that story. He started with the happy path, then came up with good boundary and edge cases for regression. Again, we stopped him after 5 minutes.

The next part of the interview was the one we were most nervous about. We had prepared a straightforward automated GUI test script in the tool we use, Canoo WebTest. We walked the candidate through the actual GUI to show what we were testing, and walked through the test. We had included in the test one module and one variable, to show that the tool supports these concepts. The rest of the script was all hard coded and included a lot of duplication, which we hoped was obvious. We asked the candidate how he might refactor the test to make it easier to maintain. He immediately grasped that the tool allowed extracting duplication into modules, and had good insight into where variables would work better than hard-coded values. He even came up with an idea to drive the test with different input values. We felt comfortable that he has a good understanding of basic design patterns and principles, such as Don’t Repeat Yourself.

Finally, I paired with the candidate to deploy a test script. Our plan was to have him read the instructions on our wiki for this, and try to do it himself, but we were short of time, so I talked him through it. First we showed him our Jenkins build UI, and the terminal window for the server where we would deploy. We asked how he might get the file to deploy, and he suggested FTP, which was a good answer. We actually copy the link for the file in the browser and use wget, so we talked him through that. It was obvious that he was well-versed in Unix commands, as he easily navigated to the right directory, renamed the file as we requested, even set his editor to vi so he could edit previous commands. We had no doubt that he is capable of learning fast, and can maintain his own test environments.

At the end of the two-hour interview, our candidate commented (un-asked!) on how much he liked our interview process. He liked the group setting, and he enjoyed the various activities we had him do. It was awesome to get this unexpected feedback.

We did our own little retrospective later, and felt we should have prepared better – we wasted some interview time in setting up the laptop and projector. We also need to watch the time better, and stick more to our planned agenda in terms of how much time to spend on each part of the interview. We all rated the candidate on some various aspects, such as his technical skills, his requirements-gathering skills, whether he seems like a good fit. He got a big thumbs-up from everyone, so I hope we’ll be able to hire him!

I got long-winded on this, sorry. If you have questions on any of this, feel free to email me. I have a lot of resources that we gathered in our effort to develop a better interview process.

Appendix: Here is the job ad we ended up with, thanks to Elisabeth Hendrickson‘s help.

Established Software Company with Startup Energy Seeks Software Tester

We provide internet-based 401(k) services to small companies. We’re part of Paychex, an established, solid, stable company, but we still have the close-knit feel of a small startup.

We’re looking for a few good people to round out our team.

What’s in it for you:

·     Work with a dials-to-11 agile team. Our developers do test-driven development, refactoring, continuous integration, acceptance test-driven development. We value interactions over processes and tools, and working software over comprehensive documentation. And we work at a sustainable pace. Lisa Crispin is a tester on the team. If you’ve ever heard her talk about the great team she works with in any of her conference presentations, this is that team.

·     Commitment to ongoing professional development. We are committed to the ongoing professional development of our team members. This is a great opportunity for anyone who values learning. We send our people to conferences and training, and support learning at work.

·     Dynamic and fun work environment. We recognize that great software comes from happy people. We have a great team environment that fosters a sense of fun and joy in our work.

We are more interested in your passion for delivering value to customers than in your formal qualifications. However, to become a software tester at ePlan Services, you will need:

·     Experience with various software testing techniques.

·     Programming or scripting skill in at least one languagesuch as Java, C, shell script, or Ruby.

·     Some experience with databases, data modeling, and SQL.

Interested? Please send your resume with a short cover letter telling us why you think this position is a good fit for you to:

The first day of Agile Coach Camp began with each person introducing him- or herself and his-or-her superpower. There were some clever ones. I said mine was getting people to write on the whiteboard, because whenever I do, magic happens, but then Tim Ottinger tweeted his disappointment that my superpower didn’t involve donkeys.

The first open space session I attended was on Systems Thinking, facilitated by Ken Furlong. I didn’t understand a lot of this, and there were some skeery formulas, but some folks tweeted some links to me to learn more about it.

Net-Maps

Net mapping is my biggest takeaway from ACCUS so far. Eva Schiffer, whose background is in things like working with GMOs in Ghana and not IT, created net mapping to identify different people or organizations that influence a project. You identify the players: people, departments, groups. You draw connections with color-coding between them, for example, hierarchies, funding, conflicts, adversaries, being in it together. People can do the maps together.

Stacks of checker-like objects piled on each influencer denote the amount of influence. This is a great visual. If you’re doing this with a group, rather than denote “positive” and “negative” influences, you could choose two more neutral goals, such as “stability” and “change”, so that people feel safe. Eva noted that the “average view” of who influences what doesn’t do anything for you, you have to let each person explain their viewpoint.

I’m such a fan of mind mapping, and this seems to take it to another level. I am eager to try this.

One of the participants, who is from the education profession (not IT), suggested it would work well for kids, for things like exploring peer pressure. It was so interesting to get perspectives from people outside of IT, both leading and participating in ACCUS sessions.

More than Agile

George Dinwiddie facilitated this session on what agile coaches need to know besides agile. Many skills we talked about were what Isabel Evans calls “thinking skills”, other people call them “soft skills”: facilitation, creating space for the development team, PR, team dynamics, teaching people outside the team to support the team rather than trying to “drive”. We talked about ways to help teams learn to self-0rganize: providing gentle direction, helping form the team, helping them take on achievable challenges. One tweetable quote from this session was “Trust goes in, pride goes out”.

I was lucky that our team had Mike Cohn for our manager/coach for the first year of our agile transition. First he helped us decide on our commitment to quality. We committed to delivering the best quality software that we possibly could. Mike helped us make this commitment mean something. He exhorted us every day to write code that we’d be proud to take home and show our moms. He told us over and over, “I don’t care how much you get done, or whether you meet some deadline. I only care that you produce the best quality that you can.” It took a long time for the team to trust this message, but once we did, we invested a lot of time to learning valuable practices such as test-driven development and specification by example. This investment paid off in the ensuing years, as we built up a good base of re-usable code and test code. We have always kept our technical debt at a manageable level, and as a result we have a good steady velocity.

QWAN: Providing basic agile knowledge online, for free

Olaf Lewitz explained the idea of a free interactive course to teach agile basics, called QWAN (Quality Without a Name). This effort was partly inspired by the Kahn Academy. We brainstormed the what, how and why of this community effort, and came up with both ideas and questions. Check out the photo.