Move Fast And Break Things With Zero-Tolerance For Product Errors

Since Rookout was founded 2 years ago we’ve been running fast. Real fast. We have created a product that some of our customers believe is magic. This magic lives alongside our customers’ dev, staging, and production environments. Moreover, it must work perfectly every time. So, how can we keep that magic alive while still searching for even more magic?

In the startup world everybody likes moving fast, and at the beginning the velocity is unreal. There are many models of moving fast and breaking things in startups but essentially, they’re all the same. You start with an innovative product idea, develop, play with it – alone or with customers, throw it away (or just a part of it), learn from your mistakes, rinse and repeat. Running this fast at the beginning isn’t an option, but a necessity required to provide an MVP that will bring value to customers.

Without trial and error, without failure, you can’t generate innovation and you can’t shoot for the moon. But what happens after you have a product? What happens after you have customers who want everything? They want your product to be perfect and trustworthy, yet still want you to be the innovator that runs fast and repeatedly amazes them. They want you to be the cool, hip startup while having the level of reliability of a big corporate IT that delivers without failure.

The (near) impossible duality of playing both sides

So your customers want it all. They want to be excited again and again while also maintaining stability. But it isn’t only about what your customers want. If you settle for stability, you will slowly wither away and become irrelevant both for the market and for yourself. “Yourself” in this case, is your soul, your employees, your beliefs, and everything that motivated you when you were just starting to run forward fast.

There are many books and articles that deal with this sort of dilemma. One particular article that made a real impact on me was Big Macs vs. The Naked Chef by Joel Spolsky. It describes how succumbing to a methodology can turn a gourmet restaurant into a boring burger joint. So how can you both serve Big Macs and gourmet food at the same time? Well, it is possible: you just need to find your sweet spot and set your boundaries.

Thank God for the Kids’ Menu

Let’s go ahead and stretch Joel’s restaurant metaphor. You, your wife and your kids go out to dine in a gourmet restaurant. As you scan the menu, drooling over the complex yet appetizing descriptions, your wife signals that it’s time to take a break from the diet. But what about the kids? The little rascals are screaming and chanting, “burger!”, fries!” And they certainly won’t settle for the “red, white, and blueberry bacon burger with basil aioli”. They want a dry meat patty stuffed between two buns and nothing else.

Thank God for the “Kids’ Menu”, it is always there. So boring, uneventful, and guaranteed to make your kids happy. Some may ask, how about innovating the kids’ menu? No way! Most kids want the good ol’ boring food with no surprises and lots of obligatory ketchup. The gourmet restaurant had set its boundaries straight: amazing innovative food for grown-ups and boring “no surprises” food for kids. These two must live side by side. Ask any parent, If the kids have nothing to eat, there’s no way for the parents to enjoy their meal. The chef’s soul is alive and well in the restaurant, but he must ignore his adventurous spirit when he prepares the kids’ menu. He does this because he knows that he must make compromises in order to innovate.

Finding your sweet spot and setting boundaries

When you want to keep running fast and breaking things while still committing to your customers and to delivering a stable product, the first thing you need to do is to map your product for its sweet spot. The sweet spot is that area where you can feel free to run fast without any worries. That’s the playground with padded walls and soft grass; you can fall, you can trip but nothing will hurt you too much. Once you’ve found that sweet spot, you can set your boundaries between that sweet spot and the zero-tolerance spots. The zero-tolerance spots are the playground with nails and broken glass scattered on the ground, and the high, steep ladders; you can’t risk a fall, any mistake made here will hurt — this is the Kids’ Menu.

Setting the boundaries at Rookout

So where is Rookout’s sweet spot and where are our boundaries? Our architecture is quite common and it is mainly divided into 3 parts.

1. SDK – That is the part our customers integrate into their code. We call this code a “Rook”. This code is responsible for creating the magic of placing non-breaking breakpoints and retrieving live data on demand.
2. Backend – Receives the data from the Rooks, processes it and sends it wherever our customers need it. It also processes the commands you’ve sent to the Rooks – whether you’ve created a new breakpoint, set a conditional to fetch your data or even requested to collect a specific variable. It also does all the user management, back-office and all the good stuff backend usually does.
3. Frontend – Rookout’s web application. This is what the user sees when using Rookout. Users can set a breakpoint, view their code, edit their workspaces, and more.

So which one of these is the sweet spot? Where can we run fast and break things?

The SDK is our temple – it is running alongside our customers’ code, whether it’s production environment, staging or dev. No matter what, this cannot break. Our customers trust us that we won’t affect their performance and thus breaking anything here isn’t an option.

The Backend must be robust and must self-heal when broken. The SDK depends on it to send data to our customers. We can run fast and break stuff but with great caution. A local failure can be transparent and indistinguishable as long as we use the right mechanism in place to recover from it. Using load balancers, multiple instances and proper fallback can actually allow us to experiment and innovate here.

The Frontend is where we can run really fast and break things. This is our sweet spot. We can make constant changes, change the user experience and create a user interface for hidden features. We release about up to 4 frontend versions per day with our amazing Jenkins setup, use LaunchDarkly’s feature flags for fast experimentations and A/B testing, learn fast from our users’ sessions with LogRocket, and other helpful services such as BugSnag, and Segment. You can read more about our monitoring tools in this blog post.

We feel free to break things in the frontend because, well, we can update versions in an instant and we have monitoring that quickly tells us when something just broke. A good investment in setting up all these tools is priceless, our developers can sleep quietly and peacefully while knowing that all the sensors will let them know if something is broken. And trust me, if something is broken, OpsGenie will wake them up.

IDE Plugin – The new sweet spot

When talking to our customers we’ve identified a new need. All of them are quite satisfied with our web application, but they sometimes want to set a breakpoint, see Rookout data or edit a breakpoint from their own day-to-day tools. The main tool that developers use is, of course, their IDE. The demand to develop a plugin for IDEs has been gradually increasing as more and more customers started using Rookout often in their daily dev workflows. Some IDEs provide a lot of documentation on developing debugger plugins, but others lack this information. Either way, these are uncharted waters for Rookout’s R&D. And uncharted waters that our customers want us to sail into are an excellent opportunity to experiment, run fast and break stuff.

Developing the plugin

We dove in head first into developing a plugin for IntelliJ IDEA and pyCharm since many of our customers use it. We wanted to experience what it was like to control Rookout outside of our web application. At the same time, we wanted to give value to our customers in order to receive feedback on it. We understood pretty fast that creating a full-blown debugger plugin with the same capabilities as our web application will take a very long time; but will it be worth it?

The IDE plugin wasn’t an existing product so we decided to do it as quick and dirty as possible. After a couple of weeks of development we created an MVP, and we know it can break, but that is fine – we want feedback and we want to move fast. When loading it for the first time we got a NullPointerException. We don’t know why at the moment, but hey… it works. Our users can install the plugin, enable it, login to Rookout via the IDE, set breakpoints, view messages and be completely in sync with our web application. And of course it doesn’t yet support all the IDE versions, but on the right versions you’ll be able to remote debug with Java and Python.

So, if you’re feeling like you want to experience our adventures in IDE plugins just go ahead and install the Rookout plugin on your IntelliJ and PyCharm. It isn’t perfect, but it is there for you and for us to play and enjoy. Want a perfect IDE plugin? Go ahead and order from the Kids’ Menu. 😉