I’ve worked primarily in a tester and quality engineering role for a few decades now. “Metrics” have always been the magic beans of improving quality. People ask me “What metrics should we use?”, as if they will reach their quality goals by measuring the right things. They want something easy, like code coverage and bug counts.
Small experiments
In my experience, it works best the other way. My teams worked to achieve one improvement goal at a time. Having agreed on the next goal, we tried experiments to move towards it step-by-step. Then we’d figure out some way to measure whether each experiment was working. For example, one team I was on wanted to shorten our cycle time, from when we started working on a user story to when we got it into production.
We decided to try example mapping to get better shared understanding of each story in the planning stage. This practice should help us cut down on re-work when acceptance testing found issues. What metrics would track the success of our experiment? Besides cycle time, we could measure the percentage of stories rejected in acceptance testing. When it was reduced by half in two iterations, we kept up the example mapping.
Measuring progress in your AI journey
Today, we have AI tools to help us achieve our quality goals. The principles of building quality in throughout the whole development life cycle remain the same. AI assistance gives us new software tools that help us work smarter. Research shows that AI amplifies the strengths of high-performing organizations, and the problems of dysfunctional ones.
In my view, approaching AI adoption step-by-step, iterating through small improvement experiments, is a good way to enjoy the benefits and avoid the pitfalls. Measuring appropriately to track each experiment is key.
Enter the DORA AI Capabilities Model. I encourage you to go download the DORA AI Capabilities Model Report. Open it in your PDF viewer of choice. Now, do a “find” on the word “measure”. Found on 23 pages! Wow! Go back to the start of the document. Check out the seven key capabilities on page 4.
Now read page 5, where you will learn that the report gives you details on how to get started on each of the seven capabilities. Keep reading. As you explore the detailed section for each capability, and why that capability matters for AI, you’ll learn how to improve that capability day-by-day. And, hooray, the report explains specific ways to measure your progress. You’ll also find the common obstacles to successfully adopting capability.
There’s a lot to digest here. You’ve got so much information here for your team to discuss. But where do you start? We’re only human, we can’t grow seven capabilities at once.
No worries, the report provides detailed sections on techniques to help your team prioritize which capability to work on first. This takes an investment of time, for sure. Then your team will be ready to start iterating on those small experiments to improve the highest-priority capability. The best part is you will now know some good ways to measure your progress towards improving that capability.
Making the investment
I know this sounds overwhelming. Most people do look pretty overwhelmed by the challenges that AI brings along with its benefits. We humans tend to not make decisions based on scientific research. Yet the potential rewards of successful AI adoption, and the potential disasters of unsuccessful AI adoption, should motivate this investment. So now, go download DORA’s ROI of AI-Assisted Software Development report. It shows the initial investment needed to reap the eventual exponential growth that AI assistance enables.
Be patient, take baby steps, do one or two small experiments at a time. Make them short enough so that you know in a week or two if the experiment is working – based on those appropriate metrics you chose. At some point, your team will get past “J-curve of AI value realization” and start saving time. Rather, making time for humans to do what humans do best, while the robots do our grunt work. As the ROI of AI-Assisted Software Development report concludes,
Invest in engineering excellence today to ensure AI acts as an amplifier of value rather than a catalyst for downstream chaos.
You’ve got the recipe books here, thanks to DORA. Invest the time now, and enjoy tracking your quality – and your enjoyment – improving faster and faster over time.
