Code Deployment Bottlenecks And How To Tame Them

If you take a long hard look at the DevOps movement, you will find it actually divides neatly into two sub-movements. The bigger and often noisier of the two is about technology, advocating for the latest and the greatest tools and techniques, be they Cloud, CI/CD, Serverless, or Containers. The smaller sister, however, is much different, stemming from Management Theorem and focusing more on processes.

Nothing says Management Theorem as much as ‘The Goal’ by Eliyahu M. Goldratt, that pesky little book ‘The Pheonix Project’ by Gene Kim, Kevin Behr, and George Spafford, was based on, which teaches us all about the Theory of Constraints and Bottlenecks. Today, I wanted to share with you my thoughts on the biggest bottleneck our customers and prospects are struggling with: the dreaded deployment.

How do you know Deployment is the Bottleneck?

To be blunt, deploying new code is always the bottleneck. Be it to development, staging, or production environments, you can (almost) never deploy fast enough. But you might want better proof than a comic strip (though personally, xkcd is more than enough for me ).

Let’s start by checking where work is piling up. Open your issue tracking system (Jira, Trello, or something else owned by Atlassian) and take a look – how many of your tickets are pending deployment?

Next, take a look at your team – engineers, product managers, support, and customer success. What are they worried about? What are they waiting for? If they are anything like the ones I’m working with, they are waiting for the next deployment to fix a bug or get a piece of data for them.

How do you increase capacity at the Bottleneck?

The obvious answer is to increase capacity within the bottlenecks. Some of the tested and tried approaches we have used for optimizing deployments are:

1. Building a (better) CI/CD pipeline, which ultimately makes deployments cheaper and faster, thus requiring less time and hand-holding.
2. Design your applications along the Twelve Factors, especially Disposability, making deployments and restarts faster and easier across environments.
3. Adopting various safety nets such as monitoring tools and advanced deployment and rollback procedures such as Canary Releases, which increases your risk tolerance to bad deployments.

Unfortunately, we’ve noticed that many of these action items are easier said than done. After all, there is a reason deployments have become a bottleneck, and if it was that easy to fix engineering leaders throughout the world would have squashed it long ago.

If you are building a new software product, you should definitely go for it. But for most existing (and profitable!) software products out there, this journey is likely to take years, cost decades of human time, and may not even net a positive ROI.

“You are reading this book for two reasons. First, you are a programmer. Second, you want to be a better programmer. Good. We need better programmers.”

― Robert C. Martin, Clean Code: A Handbook of Agile Software Craftsmanship

Riding the Bottleneck

At its core, the Theory of Constraints doesn’t focus so much on increasing capacity at the bottleneck, but rather maximizing the value you are getting out of the bottleneck. So how do you go about it? What we’ve learned from our experience is the following:

1. Make sure you are always deploying the most important task. Have the second important task waiting for its turn. Avoid switching halfway due to shifting priorities.
2. Make sure whatever you are deploying is truly ready for it – tested, documented, and meets all your prerequisites. Avoid switching halfway due to missing pieces.

Avoid unnecessary deployments by using tools such as WalkMe for in-app assistance, LaunchDarkly for managing features, and Rookout for understanding your code and adding logs and metrics on the fly.