Debugging Workflows Two Ways

Today, service architecture is becoming increasingly complex with the explosion of new software techniques such as microservices. Zhamak Dehghani, a principal technology consultant at ThoughtWorks, shares why organizations undergo such a transformation:

“The ones who embark on this journey have aspirations such as increasing the scale of operation, accelerating the pace of change and escaping the high cost of change. They want to grow their number of teams while enabling them to deliver value in parallel and independently of each other. They want to rapidly experiment with their business’s core capabilities and deliver value faster. They also want to escape the high cost associated with making changes to their existing monolithic systems.”

However, the performance of a system often is dependent on engineers’ ability to debug gnarly problems. The increase in complexity that comes with new microservices architectures makes debugging that much harder. In fact, some companies are considering reverting back to monoliths because of the increased difficulty of debugging, among a host of other challenges.

Luckily, there are some best practices for debugging that you can keep in mind that are specific to your organization’s architectural model: monolith, or microservices. Of course, everyone’s taste in workflows is as different as to how they take their coffee — the taste is often acquired — but these practices can help you adopt a high-level mindset that accelerates debugging in the context of your environment.

The monolithic method

Debugging in the context of a monolith is how most of us learned to debug — it’s tried and true, just like half-and-half and a spoonful of sugar in a strong cup of Folgers. Broken down on codementor and supplemented by Simple Programmer, this debugging workflow (called active debugging) goes a little like this:

1. Reproduce the bug: You can’t fix something you can’t find. Worse yet, if you can’t trick the system into reproducing the bug reliably, it’s likely you don’t really know what’s causing it. There’s something missing…  So you sit and think. And you get an idea (or two, or three!) of what could be causing the bug. 
2. Write a unit test: Next, it’s time to write a unit test that focuses on those potential problem areas. Good unit tests should be easy to write, readable, reliable, and fast. Remember, your bug is still out there! Once you’ve written your unit test, it’s time to check it. 
3. Check your hypothesis: Now, if you’re lucky, your unit test has discovered the elusive bug. But if your unit test passes, no worries. As John Sonmez from Simple Programmer writes, “Every time you write a unit test and it passes, you are eliminating possibilities. You are traversing through your debugging journey by locking and closing doors behind you as soon as you find out they are dead-ends.” Keep plugging along!
4. Teamwork makes the dream work: If you run out of options and feel trapped, it’s time to call a friend. Your colleagues may have dealt with a similar bug before, and hold the key to your freedom. Even if they don’t have previous experience with a similar issue to draw upon, fresh eyes can work wonders.
5. Write it down: You fixed the bug! Now, you need to write down your process. Make sure that if something like this happens again, you and your teammates have the information needed to fix it without so much effort. This step can also mean verifying that the fix works, and writing a regression test. Precautions like these can help prevent further issues.

Now, you might have noticed that we don’t mention pulling up your trusty debugger. With monoliths, it’s quite possible that you won’t need one, or at least you don’t need one right away. Opening the debugger right away is, according to John: “like when your car breaks down and you don’t know jack shit about cars, so you open up the hood and look for something wrong.”

If you want to use your debugger, or if the bug is too difficult to write a unit test for, then it’s time to rely on that trusty tool. But the key to the monolithic approach is critical thinking and unit tests.

The microservices method

The microservices method is similar to a latte: same basic ingredients, more complicated method of caffeination. Debugging microservices uses a lot of the same practices as with debugging a monolith, but with one major exception: the debugger is almost always necessary.

First of all, the many communicating parts of microservices make it much harder to reproduce a bug because it’s difficult to trace it back to the source. Additionally, logging is decentralized and harder to decipher, making active debugging a massive chore. As Rookout team member Maor Rudick writes in an article about production debugging, “This is not only time-consuming, as you will have to access and sift through many log files, but it’s also often necessary to write additional logs, and then redeploy and restart your application, just to get additional data.”

So it’s time to utilize passive debugging. According to Daniel Bryant, “The advantages of this approach is that debugging can be unobtrusive (i.e. a user’s request to an application is not paused or blocked during the debugging process), and the cycle of hypothesis identification and testing via setting multiple breakpoints can be rapid.”

Now that we’re doing passive debugging, what does the workflow look like? According to SREs Liz Fong-Jones and Adam Mckaig, it may look like this.

This doesn’t appear to be too different from the monolithic workflow; you still need to formulate hypotheses, create the fix, and verify that it worked. However, the formula/test hypothesis and develop solution steps will almost always require a debugger to find where the issue is. So opening up the debugger might now be your step one. If you can reliably reproduce your error without it, using the debugger tool might move to step two. However, it will be extremely difficult to move on to writing unit tests without it.

So has the traditional workflow been totally abandoned? Not at all. Developers comfortable with the monolithic method can relax a little. The transition to microservices won’t turn debugging on its head, but it will require an alteration to your workflow. Certainly the communicating parts of a microservice-based system complicate debugging, but your workflow can remain clean with the addition of a debugger early in the process.

So, how do you take yours?

Like trying someone else’s coffee, debugging with an unfamiliar workflow is something most people tend to avoid. You like it your way, and you’re comfortable with what is familiar.

Debugging microservices is difficult — it requires patience in working through problems that could be caused by numerous factors, and often takes testing incremental changes. However, as outlined above, there are steps you can take to adopt a more systematic approach by upholding processes. Of course, it’s important to find your own workflow based on your architectural model and tooling.

Finally, share your debugging fixes with your coworkers! While your workflows may differ slightly, documentation is always essential so you and your team spend less time wading through similar defects in the future, freeing up time for the things you enjoy.