Agile Testing with Lisa Crispin
Providing Practical Agile Testing Guidance

The Starship Enterprise was usually in control…

Recently I listened to one of Amitai Schlair’s excellent Agile in 3 Minutes podcasts (also available on iTunes), about Control. We had a brief Twitter conversation about it with Karina B., aka @GertieGamer. Amitai tweeted, “We can’t control what happens to us, but we control how we’d like to feel about it next time + what we do about it.” Karina tweeted, “the illusion of control is a fun magic trick that always leaves people wanting more”.

My dressage trainer, Elaine Marion, with Flynn

My dressage trainer, Elaine Marion, with Flynn

I enjoy the illusion of being in control. I think that’s one reason that one of my equestrian sports of choice is dressage. If I bombed around a cross-country jumping course on horseback, I’d have to let the horse make many of the decisions. The dressage arena feels so much more genteel. If I’m in perfect communion with my mount, performing the well-defined movements, I feel like I’m in charge. It’s a nice illusion! (In truth, I should be allowing the horse to perform correctly…)

The “What-Ifs”

I’ve been driving my miniature donkeys for many years. We’ve learned so much from my donkey trainer, Tom Mowery, and I trust my boys to take care of me. Still, if I’m driving my wagon with my donkey team down the road, and a huge RV motors towards us, I start getting what Tom calls the “what-ifs”. What if they run into the path of the RV? What if they spook and run into the ditch? I have an even worse case of the “what-ifs” with my newest donkey, a standard jenny, who is still a beginner at pulling a cart. She doesn’t steer reliably yet. My illusion of control is easily dispelled. What if she runs into that fence? This happens to software teams as well. What if we missed some giant bug? What if this isn’t the right feature?

We don’t need control – we need trust in our skills

Enjoying the view with Marsela

Enjoying the view and forgetting the “What-ifs” with Marsela

Tom’s words of wisdom about my worry over losing control are: “Lisa, if anything goes wrong, you have the skills to deal with it.” This is true, and I keep practicing those skills so they’ll be ready when I need them. If my donkeys run towards either an RV or a ditch, I can remember that they are actually trained, and cue them to change course and do something safer.

It’s the same with software development. We constantly learn new skills so that we can deal with whatever new obstacles get in our way. We identify problems using retrospectives, and we try small experiments to chip away at those problems. As a team and personally, if I am confident that if we have good skills and tools at our disposal, we don’t need an illusion of control. Whatever happens next, my team and I are in a position to do something about it.

In his podcast, Amitai suggests that if we can accept feeling less in control, we might make better decisions. I think if we focus on continually learning new skills and tools, our confidence in our ability to adapt to the current situation is much more important than feeling in control.


Lean Coffee

Janet and I facilitate Agile Testing Days Lean Coffee each morning, great way to start!

Agile Testing Days 2015 is coming up November 9-12 in Potsdam, Germany. This year’s theme: “Embracing Agile for a Competitive Edge – Establish leadership by delivering early, rapid & iterative application releases”. To paraphrase Elisabeth Hendrickson, the ability to release business value frequently at a sustainable pace is the very definition of “agile”. That sustainable pace part – that’s where the focus on quality comes in.

We’ve always known we can’t “test quality in” (though that doesn’t stop many companies from trying). But it can be hard to learn the practices that help us bake quality into a software product, starting when that product or feature is just a half-baked idea.

At Agile Testing Days, Janet Gregory and I will lead a tutorial on ways that teams can

We draw and brainstorm a lot!

We draw and brainstorm a lot!

build and share expertise in testing and other areas so that the whole team achieves a level of quality that lets them deliver frequently and fast. The tutorial is open to everyone who wants their teams to achieve frequent, even continuous, delivery of valuable software: testers, BAs, programmers, architects, POs, PMs, managers, Ops practitioners and more.

We hope you will come to Agile Testing Days, and if you do, we hope you will participate in our tutorial! We expect participants will push the boundaries and generate some new ideas, as in our tutorial last year where we identified ways to overcome future challenges in agile testing. (I’ve been using some of those myself, such as the rotating group brainwriting/drawing/mapping, and cube-shaped testers.) This year, we’ll again let participants shape the agenda, with these goals in mind:

  • Lists of skills testers have that they can share with the team – and effective ways to do

    You’ll take away experiments to try with your team.


  • How you can build your T-shaped skill sets and contribute to the cube-shaped team skills
  • Ways the whole team can collaborate to make the best use of test automation
  • And the goals that you bring!
Super Agile Person

Sightings of Super Agile Person are expected!

Please watch our video for more information. Hope to see you in Potsdam!


I love Weekend Testing and the chance it gives to collaborate with and learn from testing practitioners around the world. However, weekends are my only time to play with my donkeys and catch up with friends and family, so I rarely have time. In past years, I was able to participate in Weeknight Testing Europe, because it happened during my lunchtime, but AFAIK that has kind of died out. However, the first weekend in September was a three day weekend for us in the USA, which gave me the extra time I needed! I followed the instructions from our facilitator, JeanAnn Harrison (whom I’m proud to say contributed to More Agile Testing), which had me frantically draining my iPhone at the last minute by leaving on the LED flashlight plus every app I could open.

Weekend Testing is conducted via Skype chat. The facilitator provides the information about which app to test, what resources are needed (in this case, a mobile device with its battery below 15%), and other information such as a particular charter if applicable. In this case we were to test the Weather Channel phone app, with a focus on usability. The normal format I’ve experienced is one hour of testing followed by one hour of debrief, but this session didn’t follow that format.

Pairing FTW!

I prefer pairing for Weekend Testing, it’s more fun and I learn more. Shanmuga Raja and I agreed to pair, and I suggested we collaborate via a mind map. We didn’t plan that ahead, so we scrambled around and settled on Mind42. I have to say it isn’t my favorite, I prefer MindMeister (paid) and MindMup (free).

Snippet of our mind map

Snippet of our mind map

Normally I like to use the mind map to plan testing and record results. In this case, though, JeanAnn had already set out a long list of questions to guide our testing, mostly around the usability of the app, and other considerations such as how it affected battery drain. I admit to getting a bit flustered and not following my normal approach. If you click on the full mind map, you’ll see it’s a bit stream-of-consciousness. We (mostly me) recorded our reactions to the usability of the app, what we discovered as we went along, the things that puzzled us, and other information such as the devices we used.

Shanmuga’s Android phone crashed, apparently due to sudden battery drain, which was interesting. My iPhone 6 just steadily lost battery, but it didn’t seem abnormal. Having different devices made pairing especially interesting in this session, and it was fun to compare notes via the mind map.


I learned plenty from the Skype discussion. For example, I had never really thought about testing to see what an app does to the battery drain rate, though I’ve heard our own mobile developers discuss it. I also learned from JeanAnn that an app can affect the heat level of a device, I had never considered the potential issues from that.

One thing I’m working on right now is learning better ways to help other people learn testing skills. I find it hard to articulate to someone else how I do investigative or exploratory testing. So it was helpful to read the ongoing Skype discussion with comments from JeanAnn, Michael Bolton and others. This inspired me to try a charter, using the format I learned from Elisabeth Hendrickson’s book Explore It!

Explore the different settings for temperature and other metrics to discover whether they display consistently everywhere in the app.

I found an issue with the “hybrid” setting, which was supposed to give Centigrade temperatures but “anglo-saxon” measurements (eg., miles rather than kilometers). After changing to that setting, the app still showed Farenheit temperatures on some views.


After this session, I started using more charters at work, which I’ve found helpful to think about where to focus my testing and what risks could affect our new features. I’m continuing my reading to learn how to transfer testing skills to other team members. On my team, we are hoping to experiment with more tester – programmer pairing and having the developer pairs do more exploratory testing on each story before they pronounce it ‘finished’. Mike Talks has been helpful to me, with his blog posts on learning exploratory testing, and reading suggestions such as “Your Deceptive Mind“. All these ideas and techniques have been around for years, I just haven’t taken the time to make better use of them.

I highly recommend joining a Weekend Testing session. It’s a great way to hone your testing and thinking skills. Just like so many other professions, we need to practice our craft, and it’s harder to practice at work. The inspiration you can get from these sessions is priceless!

Warning – really long! I learned so much!

It was a hectic week for me. I co-chaired the Testing & Quality Track with Markus Gärtner, which meant I went to a lot of sessions in our track – no hardship! I also did an Agile Boot Camp session on “Everyone Owns Quality” with Janet Gregory, and a hands-on manual UI testing workshop with Emma Armstrong. (Slides for both those sessions are on the Agile 2015 Program). Here are some highlights of what I learned in sessions and conversations.

Note: Slides for many of these sessions are accessible via the Agile 2015 program online, I urge you to go look!

A Magic Carpet Ride: A business perspective on DevOps, Em Campbell-Pretty

Em Campbell-Pretty shared her experiences with improving quality in a giant enterprise data warehouse. I love her “just do it” attitude in the face of seemingly unsolvable problems (such as 400 pages of requirements!) I suspect her wonderful sense of humor helped her cope! Here’s a sample: “It would be fair to say that panic set in. Ah – opportunity!”

An example of Em’s creative approach: A program-wide retrospective helped stop the finger-pointing and led to improvements, but the Ops and Integration & Build teams were kept separate from the delivery team. When Em couldn’t get this changed, she appointed herself Product Owner of those teams.

Em noted that data warehouse developers often don’t come from the world of software engineering, so they don’t have the same understanding of source code control and the idea of frequent commits and integrations.

Team building with a haka

Team building with a haka

Some great tips from Em: Shrink the change, don’t try to boil the ocean. Make sure every team has time carved out to get familiar with a new process. Culture matters! Their onshore team didn’t even know each others’ names, plus there was an offshore team. Em started a “Unity Hour” to kick off  every sprint by bringing everyone together to play games. She showed a hilarious video of a haka performed by one of the teams.

The results Em shared showed that all these efforts paid off, for example, huge reductions in time and effort to deploy a new release.

At this point, I had to leave to enjoy talking with the inimitable Howard Sublett for an Agile Amped video podcast (Subscribe on iTunes to the series – those videos are full of great people and great ideas!). You can find links to Em’s slides on the program.

Mob exploratory testing! with Maaret Pyhäjärvi

Maaret Pyhäjärvi led an enthusiastic group through a mob “deep testing” session. This is a terrific way to help people learn effective in-depth testing techniques and provide quick feedback.

Use visuals to plan your day. Maaret uses a 2×2 model. Personally I like mind maps. Find what helps your brain the most. Plan to test fewer things, taking your time, going deep.

Our attempt at hands-on mob testing a real app showed this is a skill you have to practice. Mob testing follows mob programming practices. The navigator has to tell the driver – the person at the keyboard – what to do. The rest of the mob contributes their observations and ideas. Use a timer to have everyone switch roles at a set, short interval. Skills such as how to talk to the driver and where to focus your testing take deliberate practice. Take time to learn!

Exploratory Testing in a Nutshell from Maaret

Exploratory Testing in a Nutshell from Maaret

One important lesson was to ask questions, learn the terminology, and focus on the areas where feedback will be most valuable. My own team has tried mob testing once so far, and having the PO in the room helped a lot. Maaret says learning to test is like learning to drive a car. Don’t try to change gears in the middle of a turn – keep it simple, do one thing at a time, practice in a safe place.

Prototyping – Iterating Your Way to Glory Melissa Perri and Josh Wexler

In this session I learned more about how to design and test paper prototypes. If you use high-fidelity mockups on a computer, even if you created them quickly with a design tool, people fall into the “fear of killing your baby” syndrome. So much energy has gone into the idea, nobody wants to suggest changes. So, use pencil and paper. Color just distracts people – “I don’t like that blue”. Keep it simple.

One aha moment for me was the idea of creating goals for the persona you’re using. What would the user want to accomplish with the UI? We practiced sketching our prototypes for a mobile UI following a detailed narrative tailored to the persona, then role playing the persona and getting feedback on the design. Learning to ask open-ended questions is a skill I hope to improve. “What would you expect to happen when you click this button?” With these fast feedback cycle, we can learn so much before we write a line of code, and save a ton of time. Emma and I made this point in our own workshop.

Linda Rising, Stalwart Session

My sketch notes for this session are already posted. But in case you can’t interpret my drawings, here are a few highlights of the conversation.

Linda listens

Linda listens

Linda talked about how we resist change. For example, academics don’t want to modernize their programming courses: “The Cobol course is already in the can”.

Linda cited examples from Menlo Innovations on how to find joy through learning. She urged us to be our own scientists and do small experiments. There’s no one right answer, but patterns help. You’ve got to see it in action, don’t depend on “experts” who don’t know what feedback you’ll get.

Transparency is important in growing belief from the ground up. If there’s no trust, controlling doesn’t work. Don’t try to “sell” – instead, listen respectfully to what people have to say. You can change the environment. I’ve used the patterns from Linda and Mary Lynn Manns’ More Fearless Change, and I can attest to their helpfulness in trying small experiments and nudging baby steps of change.

Linda urged us to never give up on anybody. Don’t stereotype. Try, experiment, listen and learn, adapt. Fear less! We all come with biases and beliefs, but we have to continually change, people do shift beliefs. Linda recommended a podcast series and book, “You are not so smart”. That’s going on my podcast list!

Does the role of tester still exist? Juan Gabardini

Juan says we no longer identify ourselves by the activities we perform. I’d like that to be true, I do think we are moving that direction, but it seems hard for humans to dispense with labels. There are people out there who think “Testing isn’t technical, anyone can test”. Juan led us through considering “old school” testing in phased and gated environments versus how we test in agile development.

Skills for testing on agile teams

Skills for testing on agile teams

“Keep your eyes on the stars and your feet on the ground” urged Juan. Domain knowledge from contact to sell, from order to delivery is essential. He talked about “improving the last mile” by applying the scientific method. Use models such as Cynefin to understand the complexity of a feature. In our table groups, we came up with our own ideas for skills and characteristics that help with testing on agile projects.

Elisabeth Hendrickson, Stalwart

I was in the fishbowl for part of this session, which impacted my note-taking. Elisabeth shared how she moved from the title of “Director of Quality Engineering” to “Director of Engineering” by collaborating with four other directors so that they shared responsibility for global quality across all the teams working on the product, and removing obstacles for the teams. If developers have a problem, they ask the first director they see for help.

Elisabeth described how a few “explorers” joined delivery teams without testers and amped up their exploratory testing. They applied good development practices to the cloud. They automated solutions to lots of problems, for example, they use images to make every dev and test environment consistent. A message I got from this session was that we add value by providing something nobody else can provide.

One issue raised in the fishbowl was QA managers who resist a whole-team approach and don’t want to test early and often. Elisabeth said some of these managers identify as “the person at the end”, and it’s hard for them to let go of that role. We have to create safety for testers who are worried about changes.

A great quote from a participant: “Our CEO went to a conference and came back and said we should be more DevOps-y.” I didn’t note down Elisabeth’s answer to that!

Mob Programming – Jason Kerney

I don’t have good notes on this because I couldn’t hear the presenter very well. Luckily, this was recorded on video, and you can read the experience report paper that goes along with it! I appreciated hearing the experiences of Jason’s team. The paper is really helpful, with ideas such as doing study sessions. I wish my team would try mob programming for certain situations, such as when they’re doing something new where maybe only one developer has experience. We’re sure going to keep doing mob testing sessions when appropriate.

Be Brave! Try an Experiment! Linda Rising

I missed the beginning of this awesome session due to another video interview (this one along with Janet, by Craig Smith of InfoQ, I don’t know where to find that yet). But her slides are on the Agile 2015 program.

I learned the idea of continually trying small experiments from Linda years ago – it works! Linda noted that we aren’t doing scientific experiments, rather, we are doing trials, tinkering to see what works better. But, just for ease of conversation, we can still call them experiments.

Experiment? Trial? Tinker?

Experiment? Trial? Tinker?

We’re all born as natural scientists, but the way our schools work changes our focus from exploring and trying things – thinking “how” – to linear thinking, focusing on “what”. Agile lets us be babies again. Failure is OK! We can be more scientific. We need MANY experiments, and we don’t have the resources to do good science. Action is our best hope – not to find the truth or understand the why, but learn what works for us in our environment.

Linda explained confirmation bias – we only see information that confirms the beliefs we hold already. We also suffer cognitive dissonance, two disconfirming beliefs. To help overcome biases all humans have, talk out loud, draw on whiteboards, and get enough sleep. There was more, but I couldn’t sketch note fast enough.

Her message was clear, though. What can we do, to take some small action, to see whether it works for us or not? Not in theory, but really trying it. Do many small, simple, fast and frugal trials. Vary contexts, # participants, degree of enthusiasm, kind of project. Goal is learning about the thing, not proving it works for everyone. Re-test.

Be prepared to be surprised and learn even from “failure”. Don’t keep doing something because you’ve got so much invested in it. The investment is already gone. If you can’t finish your expensive steak, its cost makes no difference now. Involve everyone. Poke, sense, respond. Don’t look for answers, find ideas for trials.

If we have good outcomes, but “those people” (for example, managers) are in our way, don’t try to convince them. They also have biases supporting their beliefs. Data doesn’t convince people. Instead, show management the strategic value in a way that is easy to understand. Emphasize that learning happens regardless of outcome.

Performance Testing in Agile Contexts, Eric Proegler

Eric started by explaining some performance risks in areas such as capacity and reliability, and talked about the classes of bugs related to these. Please see his slides for this information – and for the cool photos from 1970s IBM sales literature. Most people do simulation testing right before release – when it’s too late to do anything about it.

Whole Team Approach, Circa 1970s

Whole Team Approach, Circa 1970s

My main takeaway is that we don’t have to do a perfect performance test where we have an environment exactly like production, and a load of “users” who behave exactly as real users behave. Eric called these the “illusion of realism”. Stop trying to test exactly what a user might do in a system that doesn’t exist yet. Instead ask – What’s a test I can do now, a test I can do in a day?

Small, fast, inexpensive tests done frequently as we iterate on our products will help us sniff out all kinds of performance issues. Don’t worry about imitating the prod environment. Isolation is more important than ‘real’. These tests need to be repeatable and reliable. Use simple workflows that are easy to recreate, avoid data caching effects. Be ready to troubleshoot when you find anomalies. We can build trust in these tests, recalibrate as necessary, use yesterday’s build to verify today’s results. Add these tests to your CI.

“Who needs ‘real’? Let’s find problems!” Do burst loads with no ramp, no think time, no pacing. Do 10 threads, 10 iterations each, 1 thread, 100 iterations. Do soak tests – 1 or 10 threads running for hours or days.

We can also use sapient techniques, something as simple as using a stopwatch, screen captures, videos, tools like Fiddler. One technique which we use on our own team is to put a load on the system in the background (we use Postman scripts driving our API for this) while just one or two people use the UI. You can use browser dev tools to measure performance.

You can also extend your existing automation. Add timers and log response times in your automated regression tests. This is something I’d like our team to try, since our tests are timed in CI. Watch for trends, be ready to drill down. Automation extends your senses, but doesn’t replace them.

You don’t have to test the whole system end to end to get valuable feedback. Login/logoff are expensive, and matter for everyone. Same with search functions. So focus on those. Think about user activities like road systems – are there highways?

Test at the API level. My teammate JoEllen Carter wrote Postman scripts to test API endpoints. She included assertions for the response time, and fails the test if the expected time is exceeded. Eric suggests using mocking stubs, service virtualization, built-in abstraction points to simulate in-progress.

Eric also suggested testing layers of the system, such as load testing web services directly. a test harness to measure response time to push and pull a message in a message bus. Use mocks and stubs, built-in abstraction points.

You can still do simulations before release, but in an agile project with short iterations, it pays to get feedback on performance with small, cheap, quick, repeatable, trustworthy tests.

Visual Testing Mike Lyles

I mostly took pictures of Mike’s slides, so you should just look at his slide deck. Mike had all these cool examples of how our brains get confused and misled. Understanding how our brain works can help us see what’s hard to notice, and get over our biases such as inattentional blindness.

Brain Facts

Brain Facts

Our team (again thanks to JoEllen) has started using a visual diffing tool, Depicted, to help us identify subtle changes in UIs. We hope to use SauceLabs to get screenshots from each browser/version, and run those screens through Depicted to compare with gold masters and flag differences for further investigation.

Example Mapping, Matt Wynne

I missed this session AND the open jam session Matt did. But Markus Gärtner, who co-chaired the testing track with me, explained what he learned about example mapping from attending the session. I also took a look at Matt’s slide deck from an earlier conference that is on the same theme. It sounds to me like another way to have a story-level conversation among testers, programmers, product owners and other stakeholders to elicit examples, distinguish those from business rules, and write down questions we need to answer before we can finish the story.

I also got to eavesdrop on a lunchtime conversation between Matt and Ellen Gottesdiener. Ellen explained techniques from her book with Mary Gorman, Discover to Deliver, such as structured conversations. I don’t know what they concluded, but to me, it sounded like their techniques achieved similar goals, though Ellen’s book goes much deeper into many ways to help business stakeholders identify the most important business value.

I have a feeling that any conversation you have that involves writing on a whiteboard, on sticky notes, on an online mind mapping tool, or on the back of a napkin will help build shared understanding of how a feature should work. One big takeaway from Agile 2015 is that visuals help our brains learn new things. Next time you start talking to a teammate about a story, walk over to a whiteboard, or get out some paper – and invite others to join you.

Get out there and start doing your small, frugal, repeatable experiments! Please let me know how it goes!

The Agile 2015 Stalwarts session are a fishbowl type discussion with leading lights of agile. I learn so much from Linda Rising every time I hear her talk (and when I read her book with Mary Lynn Manns, More Fearless Change). Do go watch videos of Linda’s recent keynotes. Whether it’s about bonobos or the agile mindset, you will have “aha” moments.

These are my sketch notes from Linda’s session. They might only make sense to me, but I wanted to share them. If you don’t sketch note already, give it a try. I will be able to look at these even a year from now and remember what I learned.

2015-08-04 07.01.20 2015-08-04 07.01.29 2015-08-04 07.01.34 2015-08-04 07.01.40 2015-08-04 07.01.46 2015-08-04 07.01.51 2015-08-04 07.01.57 2015-08-04 07.02.03


I enjoy listening to the “Testing in the Pub” podcasts with Stephen Janaway and Dan Ashby (along with various guests). Though the episodes make me thirsty for a pint of cider, the casual but insightful conversations inspire me to learn and try different ideas. One recent episode was about being a valued team member. I was struck by their observation that one needs confidence to be an effective communicator. If you are confident in your skills and experience, you can go talk to anyone on the team to ask questions, to raise and investigate issues.

That was a real aha moment for me. As testers we talk a lot about learning about our software to gain confidence in what we’re going to deliver. But I hadn’t thought about the value of being confident in yourself. Janet Gregory and I have been presenting conference sessions about whether a tester needs programming skills to be to be useful. Our view is that technical awareness helps testers communicate with programmers because it gives them a shared language. But after hearing the Testing in the Pub podcast, I think that learning some technical skills also builds confidence.

Looking back on my own career, I remember that when I faced a tough challenge, such as learning a new tool, I felt confident that somehow I would succeed. I started out as a programmer/analyst, and though it’s been a long time since I spent a significant amount of my time coding, I feel confident in conversations with programmers. I’ve learned a bunch of difficult domains, so I believe I can learn any new domain quickly. That makes me confident in approaching business experts and learning from them.

Janet and I have six confidence-building practices to succeed with agile testing (see below). What are the confidence-building practices to succeed in adding value as a tester? Just off the top of my head:

  • Make time to learn. Set goals, make a personal kanban board or other organizing tool so that you’ll work on them, use pomodoros or some similar technique to pace yourself.
  • Public speaking is scary, but builds confidence. Helping others learn means you learn too. Volunteer to share your unique experiences at a local meetup.
  • Put a bowl of chocolates on your worktop and invite teammates to come help themselves. It gives you a chance to chat informally and get to know them better.

What ideas do you have for building your own confidence as a tester and team member?

Confidence Building Practices for Agile Testing

This is the second of possibly a few posts about my “aha moments” at Agile Roots. In addition to pairing with Janet on two sessions (see previous post), I was lucky enough to pair with Amitai Schlair (see his awesome Agile in 3 Minutes podcasts) for a DevOps Dojo.

The code’s ready. Wait, what is this data?

We asked participants to pair up, with one person playing the role of Dev, the other Ops. Our “product” was a script to print labels (see GitHub for workshop material). We began with Devs throwing their code over the proverbial wall to Ops, who provided the production data. No surprise, addresses such as “Belling, Vicar of St Loony Up the Cream Bun and Jam,Arthur,Rev.,789 Incubator-Jones Crescent” tripped up our software.

After a retrospective, the Dev and Ops pairs worked together (hey, that was the conference theme!) to drive the scripting with tests involving the “prod” data.

Feeling their pain

In hindsight our exercises were a bit simplistic, but hey, we only had 90 minutes. More important to me was that we had real-life Operations people in our workshop! I was fascinated to hear their “war stories”. One pointed out that the last person to touch the code gets the blame when things go wrong. In my experience, sometimes this is Test – but of course, if Ops deploys the code to production, they get the blame first.

Experiments to try

Ideas for Dev-Ops Pairing

Ideas for Dev-Ops Pairing

In the last section of our workshop, we brainstormed on ways each participant could go back and foster dev – ops collaboration in their own organization. One interesting idea was “monitoring-driven development”. Ops can provide helpful production metrics as well as helpful tools. This can be effective for things like improving product performance.

One Ops person said he collaborates with Dev to do Operational Release Tests on an unused prod server before doing the official prod deploy. Making a plan for easy rollbacks seems obvious, but sometimes is neglected.

We talked about the need to support each other, and build trust across the organization. Even dipping your toe in the pairing water with 10 – 15 minute Dev-Ops pairing sessions can help.

Trust often simply has to be earned. If you can show how

More ideas

More ideas

your tests keep bad problems out of production, others will pay attention. A one-team approach helps build relationships and break the chain of blame.

The importance of empathy shouldn’t be a surprise, but I hadn’t thought about Dev – Ops collaboration so much in those terms until the Ops folks in our workshop told their stories. We need a mindset shift. A “Not my problem” attitude doesn’t build relationships.

Your experiments?

If you were in our workshop, I’d love to hear if you tried any of the ideas we discussed in the workshop. If you weren’t in our workshop, I would still love to hear about anything you’ve done to build those bridges between Dev and Ops.

It’s not news, but… why aren’t we all doing this already?

The lines between testers and programmers have blurred in recent years. So have the lines between operations and other roles on the software delivery team. We may be better off thinking in terms of competencies rather than roles. Working Together ™ to guide development with tests from the Ops perspective as well as that of testers and others is, to me, the obvious way to go.


I learned so much last week at Agile Roots 2015 last week. Check out the artifacts, they’ll inspire you too! Janet Gregory and I did a plenary talk on “Do Testers Have to Code… To Be Useful?” I always love pair presenting with Janet. She did a super job of explaining our views on the subject. To summarize: Your software delivery team already has coders, and they can write test code as well as production code. But we think testers do need technical awareness to help them communicate and collaborate well with other team members.

This blog post is meant to be about our workshop, though, so on to that. We had 90 minutes and a great group of participants to think about what skills a team needs to help them build quality in to their software product. Testing isn’t a phase, as our friend Elisabeth Hendrickson so aptly says. We know we can’t test quality into a product (I am not sure who first said that, but I’ve heard it for 20 years! Still, people seem to try!) Quality has to be baked in. What skills help us do that? As testers, Janet and I tend to focus on testing skills, but are they the most important?

T-Shaped Skills

Each of us has a wide range of thinking (aka ‘soft’ or ‘people’) and technical skills. Most of us also have some area of special passion where we have deep skills. For example, I have lots of experience in exploratory testing, test automation, eliciting examples from customers, SQL, and so on. But I can bring the most value to my team with my ability to learn domains quickly – that’s my deep skill. I learned about the T-Shaped Skills concept from Rob Lambert. Each workshop participant noted their skills which can help their team build in quality, one per sticky note.

Commitment to quality

Quality is like Mom and apple pie. Ask any software delivery team, they’ll say they want to create a high-quality product. But are they really committed to doing that? What will they do when they encounter an obstacle? We shared stories and discussed the importance of making that commitment mean something. It will take a variety of skills, experience and perspectives to creatively overcome all the things that get in the way of building in quality. Get your team together and talk about what your commitment to quality really mans.

Square-Shaped Teams

When all team members put their T-shaped skill sets together, we get square-shaped teams, see Adam Knight‘s blog post on this topic. Our workshop participants compared their individual skills, grouped similar ones, and discussed which were most important. (Pictures of the results are at the end of this post). What skills can each specialty bring to the party? If an essential skill is missing, how can your team obtain it?

Transferring knowledge, effecting change

We discussed collaboration techniques teams can use to make the best use of specialized skills they need. Learning new skills or sharing specialized ones can mean change, and change is hard. Patterns from More Fearless Change by Linda Rising and Mary Lynn Manns are helpful as you try to spread new ideas or encourage new experiments.

Each workshop group discussed the skill area they deemed most important, and thought of experiments they could try with their own teams to build those skills. Interestingly, communication skills, rather than technical testing skills such as exploratory testing or test automation, were tops in three out of the five table groups. The other two groups chose related skill areas: conflict resolution and gaining empathy with users. Interesting experiments were tried. One group decided to try teaching a simple skill to see how hard it might be. One of the group members was left handed, and set about teaching the others to write left-handed. This proved a simple way to learn how to teach a skill, a pre-requisite to helping spread skills across the team! Another group played an icebreaker game to learn more about each other as a first step in improving communication. Again, this is something simple and fun that any team can try.


With only 90 minutes for our workshop, we didn’t have time to try out a lot of techniques to transfer skills. For myself, a key giveaway (I learned that term from Alex Schwarz and Fanny Pittack at last year’s Agile Testing Days, I like it better than takeaways) were that what so many play down as “soft” skills form the core strength of a team’s ability to build quality into their software. If they can’t communicate with each other or their customer effectively, it’s hard even to define what quality means to them and to their customer. Another “aha” moment was realizing that extremely simple exercises such as an icebreaker game or teaching a skill like writing left-handed provide a lot of insights and help teams work together better.

Below are the skill charts from each of our groups (WP won’t let me format these in a nicer way, for some reason). You can also check out our slides, which have some good resources for further reading. Janet and I will do a similar workshop at Agile Testing Days, but we’ll have a whole day there, so we are looking forward to more in-depth outcomes which we can share.





I work every day as a tester on the Pivotal Tracker team. Some people think that because I’ve written books and speak at conferences I must be a full time consultant, but my passion lies in being a member of a great team and doing hands-on testing.


Brainstorming with JoEllen, at our old office

It’s easy for me to go around telling you all what’s the best way to build quality into a software product, but practicing what I preach can be a challenge. For example, I’m a very shy person with self-esteem issues. So though I’m always telling you how great pairing is, I often find it hard to leave my bubble and pair with amazing people like my teammate JoEllen Carter.

Here’s a slice of life from a particularly exciting week. We moved to a fancy new office half a block down the street, along with other teams from Pivotal Labs. Right now our Tracker team is rattling around in our new digs, but we have several new hires, interns and a Colorado School of Mines project team joining us soon.

Enough said

Enough said

Until all those new folks join us, there are extra monitors lying around. One of my teammates, knowing how much I love major monitor real estate, hooked a third monitor up to my usual workstation. Of course, another teammate caught me using my laptop for a standup meeting with some remote team members, and posted it on our Slack channel. Yeah, I look pretty silly! And there’s a downside: it’d be nice to move locations every day as our dev pairs do, but I can’t tear myself away from those three monitors. Well, it won’t last forever.

On the right are the scenarios I wrote, on the left are the designer's ideas.

On the right are the scenarios I wrote, on the left are the designer’s ideas.

Our new space has acres of glorious whiteboards. We had few whiteboards at the old office, but whenever any of us got in front of a whiteboard and started drawing while talking, magic happened. Today, a few of us started discussing a poor user experience in our customer signup process. After a few minutes of waving hands and explaining, I walked over to the giant wall o’ whiteboards and wrote out three scenarios. The others walked over and we had a good conversation.

Later on, the designer and a couple of developers went back to the whiteboard to talk more about it. The designer sketched out his ideas. Writing and drawing on the whiteboard helped us think things through and share the same understanding about the problem and the potential solution.


New Tracker office

Other stuff that went on this week? I usually work from home two days a week because I live 35 miles and much gridlock away from work. The office move translated into problems connecting with the office. The team is changing up how we do builds and deploys, and that’s getting in the way of delivering stories for final acceptance testing. We testers pitch in on customer support, and that’s been a bit busy this week. Like everyone I know, I don’t have enough time to do all the things I want/need to be doing. But we sure have a pretty new office!

Oh, and I did NOT pair with JoEllen at all. This is terrible. OK, she was gone the first two days of the week. We have a three day Hackathon next week. I had better take advantage of the opportunity.



Janet Gregory and I enjoyed participating in the Quality in Agile conference in Vancouver April 20-21. We paired on a keynote: “Do testers need to code… to be useful?” Our opinion in a nutshell: testers need technical awareness to collaborate effectively with all their team members, but our software delivery teams already should have expert coders!

Even if they don’t write code, testers need to participate in automating regression tests and other useful automation, in collaboration with programmers, business stakeholders and others. Janet and I facilitated an all-day workshop on advanced topics in agile testing, with a focus on automation.

Challenges around automation

After some introductory slides, the 15 workshop participants self-organized into three smaller groups, choosing to sit with people that had similar goals for the day, or who had experience related to their goals. Each person listed their team’s impediments to automation, one per sticky note, and we grouped these on a big wall chart.

Automation Challenges

Automation Challenges

Next, everyone dot voted on the topics they wanted to tackle during the workshop. The top three vote-getters were:

  • Culture and responsibility – whose job is it to automate?
  • Lack of time for automation activities
  • Things that make tests hard to automate, such as complexity

(Note, you can find higher resolution photos of all the session wall charts, including those not on this post.)

Formulating the problem and brainstorming ideas to overcome it

Mind map and problem statement for cultural challenges

Mind map and problem statement for cultural challenges

Each group was tasked with writing their own problem statement for the culture and responsibility topic. It is challenging to write a good problem statement! You can see an example at the bottom of the mind map at left. Once the problem was defined, everyone picked up a Sharpie and each team mind mapped on their big piece of easel pad paper. IMG_4427

One group focused on a lack of shared vision and investment in automation at the company level. Another saw a lack of education on both sides.

Third group's mind map is on the right - my individual pic of it was blurred

Third group’s mind map is on the right – my individual pic of it was blurred

I thought it was interesting that the third group exploring culture and responsibility mentioned doing social activities together, and honing soft and tech skills including being respectful of each other.

For topic #2, after each group wrote their problem statement around the lack of time for automation, we tried a different brainstorming technique: Brainwriting. Each person wrote their ideas for dealing with complexity and other things that make automation difficult on a plain piece of paper. Every three minutes, they passed their paper to the group member to their right. They read what was written already, then wrote more ideas. This continued until each person within the group had written on each paper. Most people agreed that reading other peoples’ ideas jogged new ones for themselves. This technique lets people who might not be comfortable coming forward to draw on a mind map or say their ideas aloud contribute equally.


Example problem statement


Ideas for dealing with a brittle app


Ideas for good code design and for collaboration

For topic #3 (sample problem statement to the left), we did “brainwriting with a twist”, an idea of Janet’s. Each team started by drawing, mindmapping or brainwriting ideas on a big flip chart page. After 10 minutes, each group moved to the next group’s flip chart, read the problem statement and ideas, and added their own. Some specific ways to design better automation code came out of this, as well as ideas for better tester-coder collaboration and ways to make these problems more visible.

Designing experiments

Example experiments

Example experiments

Screen Shot 2015-04-26 at 1.11.02 PMOf course, there is more to problem solving than brainstorming ideas. Janet presented a model of Esther Derby’s (left): define the problem and desired outcome, understand the context and requirements around potential solutions, design experiments, try them and evaluate the results. Each team spent time coming up with experiments they will try when back with their own teams. We hope that participants will report back to us on how their experiments went!

Got automation challenges – or any challenges related to quality and testing, for that matter? Get your team together, try out some brainstorming techniques, make it comfortable and safe for each person to contribute their ideas. Identify the biggest problem, brainstorm a couple of experiments to try to make that problem smaller, and use your retrospectives to evaluate the results. Keep experimenting, inspecting and adapting. Over time, your problems will be smaller and your successes bigger. But remember to celebrate even the small successes!