Book Review: A Practical Guide to Testing in DevOps by Katrina Clokie

Many of us have followed Katrina Clokie‘s blog posts and/or had the pleasure of hearing her present at conferences. Now she has compiled not only her broad experience but that of many other practitioners into this awesome book. I’ve just finished reading it and am sharing my thoughts on it here. This post is a bit long, but there is a lot of goodness to cover in the book. And it’s my blog, so I’ll do a long post if I want to!

This is indeed a practical guidebook for the present and future of software development. From continuous delivery, to the use of machine learning and artificial intelligence for testing, and beyond, the practices in this book will lead you into the future. There are so many ideas in here that are new to me, and links to more resources to dive more deeply into them.

One idea I’d like to try I the test strategy retrospective. Visualizing the test strategy along the path to production is powerful, I’ve been making a start on that and finding it helps identify risks and generate ideas for experiments to mitigate those risks

Blazing new trails

The book gives us new ways to learn how to collaborate across disciplines. The author explains how small teams are better able to collaborate. I love the metaphors on “blazing trails” to people outside your delivery team, working towards a collaborative exchange between disciplines. She explains how to build on these trails to widen the number of people who are connected and the information flowing among them. This book embraces that agile value “Individuals and interactions over processes and tools.” We testers are good Question Askers, and the book gives us lots of ideas for questions.

An essential part of the book is ways to make testing activities visible, including specifics and helping others picture what your testing involves. In my experience, making a problem visible is the first step to solving it. Pictures help people learn. As you read, you’ll get many ideas for experiments you can try with your team, such as dojos and job rotation.

DevOps 101

The author makes concepts such as pipelines accessible to the newbie, with pointers to go learn more. She presents important insights into how to identify potential risks of feature toggles, and deal with difficulties in testing permutations of multiple toggles. There are many good tips for things we might forget like testing the production monitoring.  One new idea I got from the book is testing the design of analytics data collection. The book shows us the waves of the future too, such as using machine learning to make automated tests adaptable to changes in the app.

I learned a lot about terminology that I didn’t know I didn’t know, such as the difference between canary releases, staged rollouts and dark launches. Words matter, it is important to know what these terms really mean and how the different practices work, so you can understand the pros and cons of each. While I was familiar with a lot of these terms, I hadn’t thought of the specific meaning and differences between one practice and another. It clears up the cobwebs to have specific definitions to work with. Adding examples of things like configuration management scripts to this, I feel much better equipped to collaborate with my teammates who focus on DevOps.

The author reminds us that we may need to test infrastructure such as monitoring and logging. She encourages us to make use of feedback, both from analytics and customers using a feedback widget or emailing.

Many ideas for experiments!

I appreciate how the author presents and connects ideas so that you can see what combinations of practices you might like your team to try. For example, maybe your team is already using post-release monitoring to detect problems, but you could try adding in beta testing or pre-release exploratory or automated testing. Understand the benefits and limitations of different types of testing, in different contexts.

It’s helping me understand what our team has been building up for the past couple of years in terms of moving hosting to the cloud and changing how our test environments are created and maintained. I could work without knowing the internals, but knowing some of the internals helps me be able to have better conversations.

I was interested to read arguments for different types of practices such as eliminating a staging environment. Again, this is something my own team has done without my knowing all of the reasons behind it. I could work without the knowledge, just like I could drive a car without knowing how an internal combustion engine works, but it’s nice to connect all the dots.

Learning by example

The case studies include great examples of how to make information about build, test and deploy activities easily available and highly visible. Learning how other teams have analyzed and addressed their testing and quality risks helps me think about where my own team should focus testing. Obviously, testing in a continuous delivery environment has to evolve from what we did in the days of less frequent deploys to production. It’s hugely helpful to learn more options available to us.

Models for thinking, planning, collaborating

The book presents ideas for identifying and prioritizing risks, and coming up with a test strategy to address those risks. This includes use of models such as variants on the test automation pyramid, rather charming yet effective visualizations of bug filters, and an alternative to that, the DevOps Hourglass. I have found that using models like these helps the team get away from unconscious biases and do a better job of identifying what types of testing need to be done, what resources are needed to do them, and when they should be planned and executed. Similarly, the author presents a pendulum model for determining an exploratory testing strategy.

Future of testing and testers?

The book addresses the challenges of testing with continuous delivery. Testing isn’t on the right or left, it also has to be continuous. Each team has to talk about the level of quality to which they intend to commit, how they will measure that, and then find ways to achieve that level of quality. With continuous delivery, that means bringing in or giving more emphasis to practices such as monitoring in production. The author addresses how the tester role has evolved, and how some feel it doesn’t have a place in DevOps (they said that about Extreme Programming, too!)

I love the idea of a visual test strategy, this is definitely something I’m going to try with my team. As the author says in her conclusion, DevOps helps us think differently about testing. For me, moving towards continuous delivery in my own team is such an exciting time. This is a much-needed book to guide us into the future of software testing and delivery.

A few observations

Like many publications and presentations on DevOps, this book cites the term being coined for the first DevOps conference in 2009. But some of us – perhaps many of us – have been on teams practicing DevOps for many years, we just didn’t have that name for it. I was on a waterfall team back in the mid 1990s which did many of these practices, such as continuous integration with excellent test coverage at the unit and GUI levels, sophisticated configuration management and automated deployments to a couple dozen platforms. I was on an agile team that achieved continuous delivery in about 2005. But, hey, now we have this cool label (which I keep typing as “DevOops”) to hang on it.

The book links off to so many great resources. I found myself wishing that the author had included more of these within this book so I didn’t have to keep going elsewhere. I know the downside is that it makes the book long, and nobody likes thick books these days. And the up side is I can really get in-depth information on these different topics. But I would like to just keep reading in the book.

Keeping this handy at all times

I wish this book were available in hard copy so I could keep it next to my keyboard, along with Elisabeth Hendrickson’s Explore It!  And the books Janet and I wrote. I like to be able to refer to these for quick inspiration and testing ideas. I’ll certainly keep this one open on my laptop!