Experiment-driven quality

Wow, this is my fourth blog post sharing some of my biggest takeaways from Agile Testing Days, yay me. Some of my most inspiring and actionable ones are from Lisi Hocke’s talk, “Growing an Experiment-Driven Quality Culture”. Follow that link to see her slide deck. Often a slide deck is not much use without the accompanying talk. However, Lisi’s slides convey her main ideas well, and there’s a ton of great advice there. Here are some of my notes from her talk – hopefully, accurate enough!

Lisi and I did a workshop at Agile Testing Days 2017 on “testing in a continuous world”, where we had participants identify their biggest obstacles to testing in continuous delivery, and design experiments to overcome them. We had an enthusiastic response, and I’ve continued to use this exercise in workshops and tutorials (pairing with other awesome people!) I’ve also occasionally gotten my own teams to design experiments to address our biggest problem. It’s exciting to see how Lisi approached this on a larger scale.

I loved how Lisi approached her team’s challenges in such a thoughtful manner. They were working in a complex environment, and wanted to improve testing and quality. They did my favorite thing for chipping away at a problem – they designed experiments! With hypotheses, and measurements to track progress.

Success factors

Lisi got some critical success factors in place. She and her colleagues had clear goals – creating transparency, raising awareness, success criteria, and exit criteria. They had the support of senior management. They selected four product teams as pilot teams. And they spent time promoting effective communication (see her slides for Lisi’s excellent communication guidelines).

Pilot teams were chosen for the initial experiment. I loved the idea of a kickoff workshop where they visualize the pain points in their pipeline. They did risk storming, another favorite practice of mine (and thanks to João Proença’s excellent keynote “Limitless beyond our boundaries” (see my post on that), I know why risk storming works so well!) They followed up with a workshop to gain more knowledge and skills in necessary areas. I will try these ideas whenever I can persuade my own team to try more experiments. It’s so smart to decide on a specific desired outcome, such as adopting one new practice to try.

Lisi found ways to make it as easy as possible to experiment, choosing just one of the team’s current challenges and trying only one new practice to address it. At the end of the experiment time box, the team discussed whether that one new practice provided value in their context. Even if an experiment doesn’t achieve the desired outcome, the team still gets valuable learning.

Naturally, there were obstacles to doing the experiments. Everyday business took over quickly. Even so, they stuck to their core values, to team responsibility to quality. The continual learning helped them build confidence. The experiments generated conversations, raised awareness, gave people a chance to improve skills, and inspired teams to improve practices. On the negative side, not everyone got on board with the experiments and new concepts.

Feedback loops

One big challenge for Lisi was how to scale this process to 33 product teams. A second experiment was focused on learning teams’ testing strategies, and giving them feedback. This would inform subsequent experiments. They based the feedback on the core values of whole-team responsibility for quality, automating where possible, investing in quality and exploring the unknowns. This approach showed promise– and then the pandemic happened.

Lisi found that product teams wanted her to enable them to solve their own problems – they didn’t want to be told specifically what to do to solve them. Lisi recommends building on peoples’ curiosity to create momentum, with tech talks and personal conversations. Understanding their needs and finding helpful indicators of progress, looking at what the team already tried, is key.

I was surprised that Lisi heard this kind of attitude from teams (not from the managers!): “We don’t have time to plan and improve testing and quality. Those are a ‘nice to have’.” I have run into this before myself. The managers are bewildered because they think they are encouraging taking time to learn and improve, but the teams are still feeling too much pressure to slow down and experiment. Lisi and a colleague held a series of leadership workshops to build a foundation for quality.

An experiment-driven approach

I first learned about doing small, frugal experiments to promote change and improvements from Linda Rising. When I’ve been able to get my teams to identify the biggest obstacle to our goals, design an experiment to make the obstacle smaller, and take responsibility for actually trying out the hypothesis, the teams made huge progress in a short time. Unfortunately, as Lisi mentioned, people feel time pressure – whether it’s real or only perceived – and push back against taking the time to experiment. Lisi’s talk has renewed my motivation to do what I can to influence my teams to try the experiment-driven approach to continuous learning and improvement.