New Challenges in Continuous Delivery

Continuous delivery and deployment (both abbreviated as “CD”) aren’t new concepts. Jez Humble and David Farley published their Continuous Delivery book ten years ago. Patrick Debois organized the first “DevOpsDays” conference in 2009, and created the #DevOps (also written as devops, devOps and Devops) hashtag.

Of course, CD is new to many teams. There are lots of publications and courses to help organizations successfully adopt CD. As with any practice, once you’ve mastered the basics, there are always new problems to solve. This past January, I was fortunate to participate in DeliveryConf, a new conference providing a place for people to get deeper technical information about continuous integration (CI) and continuous delivery. Based on what I learned there, along with my own experiences and talking with people in the community, I’d like to share some of the challenges today’s teams may encounter with CD.

Testing

The DeliveryConf sessions I attended (or watched later on their YouTube channel) helped me see more ways the technology and practices around continuous delivery have changed how we test.

Automating the drudge work or toil isn’t a new idea, but Jessica Kerr’s talk gave me some new perspectives on this. Here are a few examples of how we can use automation to free our brains up for more important work:

• Automation to reduce waiting time for approval or for a machine to be available
• Automate things like procedures, tools, getting credentials, PR review process
• Spin up test environments in the cloud for each new change

We have fewer bottlenecks, less contention for shared resources, less time wasted waiting for handoffs, and much faster feedback loops. Continuous delivery keeps the software “alive” in our heads so that we are always learning about it.

Mapping your pipelines

A value stream map is a technique that comes from Lean manufacturing. It’s a way to visualize how a team delivers something of value to the customers. A value stream map shows all critical steps in the process, focusing only on the things that add value to the product. It includes the time each step in in the value stream takes, and shows time spent waiting for things that don’t add value, like handoffs, as well as waiting for people to make decisions and waiting for resources such as test environments or people with particular specialties to become available.

In his talk, Steve Pereira recommended making a value stream map of your pipeline to detect bottlenecks and shorten feedback loops. His team mapped theirs in Google Sheets so that everyone could see it. They looked at the manual steps, the automated steps, handoffs, and wait times. They worked on one doable slice at a time. Eventually, they were able to cut the time it took to run the whole pipeline down from four weeks to four days!

Here’s an example I made of what a portion of the value stream map of a pipeline can look like, including the time each step takes to execute, and waiting time in between steps:

Steve gave some great advice that I also heard in other sessions: Don’t base your own process on what some well-known company is doing – keep it relevant to your own context. Be sure to include customer support folks, marketing, sales… everyone who is affected by your deployment and release process in these mapping discussions. For example, your map should include building support materials and help docs in earlier stages, not right at the end where it might block release.

Teams that have been doing CI/CD for a while may have “legacy” pipelines – pipelines that have been around for a while. They usually aren’t well documented, nobody on the team quite remembers how they were built or understands them completely, and they might even be unstable. At DeliveryConf, Laura Santamaria shared her experiences with legacy pipelines. She also recommended making a value stream map for the pipeline, using any documentation you can find, including anything written down in the issue tracking system. This helps understand the pipeline and find ways to stabilize, improve and document it.

We need visuals to help us see what is not working, and talk about how we will improve. Value stream mapping is a tried-and-true technique, and is simple to do.

Beyond “traditional” software apps

Emily Gorcenski talked about continuous delivery for machine learning (ML), sharing patterns and pains. I have only basic knowledge about ML, so this was particularly eye-opening for me. I had not realized that ML models have a shelf life.

Emily explained that data-driven systems are non-deterministic, which makes them difficult to test and improve. They have complex acceptance criteria, are black box and are highly non-linear. Small changes can have an unexpected effect in some other areas. Boundaries are hard to define. She said, “data-driven development is hard, and we’re all kind of bad at it right now.” Long story short: CD with ML is still hard to do. This sound a bit discouraging, but it just takes more experimenting to work on those challenges.

Multiple teams may be needed to produce data, build and test the ML product, and deal with the technology transitions in between. That means potential delays so that by the time the product is in production, its insights have expired. Addressing these big challenges takes lots of collaboration across disciplines and a lot of sophisticated infrastructure.

The takeaway for me is that continuous delivery applies to a broad range of software products. CD is being more widely adopted for web-based applications, embedded software, operating system software. Applying it to other types of products such as  ML models, BI systems, video products, and audio products –  things that teams currently integrate right before they try to release. These types of products pose some of the biggest challenges for continuous delivery today, but they aren’t impossible to overcome. We can keep looking for ways to “shift left” and build quality into all types of products, as well as “shift right” and figure out ways to monitor and learn how they are used in production.

Do more than talk about it

A strong message at DeliveryConf was to “actually do CD” – one small step at a time. Focus on the most important thing to accelerate. Try small experiments, make them visible, and measure your progress. Even legacy systems can be continuously delivered – we heard from someone who had implemented CD for a 45-year-old VAX assembly application!

I was struck by the emphasis on small steps and experiments. We hear this same message at conferences that are focused on testing and on agile development. DevOps is about removing the silos within organizations. What if we worked harder to remove silos in our software communities? We all have a lot we can learn from each other. There are lots of virtual conferences happening in the DevOps space, including DivOps, Failover Conf, and All Day DevOps. Take advantage of these learning opportunities!

Automation isn’t the only important part of CD. Organizational culture and relationships among people with different skills and roles are necessary too. Again, this is a message we’ve been sharing in the testing and agile communities, it’s true for just about any activity that helps us solve our customer’s problems with a software product.

Look at what you can automate next in order to improve or move towards CD. Ideas for what to automate next can come about organically just by having open discussions about everyday challenges we face. Find ways to integrate the important activities that can’t be automated into the pipeline while still getting fast-enough feedback. Look for tools and techniques to help mitigate risks once new changes are in production.