Debugging, With A Little Help From My Friends

When I’m on pager duty and the phone rings at 2:00AM with a production crash caused by some code change that was pushed too late in the day, what I really want is a friend. Someone to tell me that it’s all going to be OK, that I’ve solved a hundred such issues, that the solution is right under my nose, that if I just fetch the right log line or stare at the right code snippet the solution will pop into my head.
If they can also brew coffee and point at the right code line, showing me where the problem is – all the better. But even if they can’t, just knowing they’re with me on the line, listening to my educated guesses and assuring me that I’m making some kind of sense would make all the difference.

Rubber duckie, you’re the one?

During office hours, when things are calm, a rubber duck may be enough. My rubber duck sits quietly on my desk, listens to me drone on and on about the changes I’m making and where I think the issues are, and never judges me. But when production is broken and the slack channel is screaming, I need a bit more than that. I need a debugging friend.
Regardless of how you feel about Pair Programming, my experience says that Pair Debugging is extremely effective, and much less frustrating. So whenever possible, I try to implement Rubber Duck debugging with a real live programmer friend.

Ideally, the conversation would go something like this:

Me: This is where I think the problem is.
Friend: Why do you think that?
Me: Well, I expected this variable to be set, but instead I get that exception.
Friend: Where could this exception come from?
Me: Well, there are three user flows that could lead to it.
Friend: Have you tried isolating which of the three is happening right now?

Having a friend who understands basic programming (even if they’re not intimately familiar with my code, or with our entire application), who knows which questions to ask, and who understands my answers well enough to know if I’m making sense, would make all the difference and keep me sane and effective enough to get to resolution quickly and happily. In fact, debugging in pairs is a powerful method for finding tough bugs more quickly than individual users can.

Creating a home base for pair debugging

Initial attempts at building a digital debugging companion have been amusing, but not necessarily effective: For one, it still feels similar to speaking to an actual rubber duck (and I don’t need an app for that). But more importantly, you’re not getting the proven benefit of having an actual programmer friend ask you intelligent, relevant questions and look at your code with a critical eye.

A more significant step was taken by the Visual Studio team, who released Visual Studio Live Share in May 2018. Visual Studio Live Share allows developers to collaboratively debug their code in real time, sharing code context while independently investigating an issue without stepping on each other’s toes.

Visual Studio Live Share works well when you are debugging locally, and when you can afford the time to perform step-by-step debugging, and if you’re a Microsoft shop — or at least a Visual Studio shop. But Pair Debugging is still needed if you’re debugging remotely (for example, debugging a remote staging environment, or a serverless or microservices deployment), and when you can’t afford to set breakpoints (for example, if you’re debugging production). That’s where Rookout comes in.

When we built Rookout, a rapid debugging solution, collaboration wasn’t at the top of our minds. We were concerned with making it easy to fetch debug messages from any environment and stream them to any log collection and analysis tool a user chooses.

Whistle while you work(space)

Recently, when we added our Workspaces capability, we realized that we’d stumbled onto something that would make devs (that is to say, us) much happier the next time we find ourselves debugging our prod environment in the wee hours of the night.

Here’s how it works:

As I start debugging, I determine the subset of environments to look at.

With Workspaces, you can easily define a subset of your deployment and source code, and share it with members of your team. When your debugging friend logs into Rookout, he can join you in the workspace and start debugging wherever you are.

When teammates join the debugging effort, they can apply the same filters I’ve already defined by joining my workspace. They will immediately see where I’ve set breakpoints, and which debug messages these breakpoints generated. This also point them at the code snippets I’m trying to debug.

Any member of the team can add their own non-breaking breakpoints into the Workspace and debug together without sharing screens or granting access to teammates’ machines.

To take this one step further, we’re considering adding a “share” option which would point a friend to the exact code snippet you’re looking at. They could ask you the right questions, direct your attention to another line of code, and do whatever a friend needs to do to help you get through an urgent production issue more efficiently than you could do on your own.

Light up the long, dark night of the soul of your app

Imagine the next time you’re woken up at 2 AM, and realize that you need to debug an urgent issue in production. As you log into Rookout, you can see where your teammates are looking. You see which code snippets they suspect, and the log lines or stack frames or error messages generated by the snippets that raised their suspicions.

You can reach out to your teammates and ask them what they’ve already seen, what they surmise, and what they’re planning to try next. And you can start your own investigation, pointing your teammates to code areas you suspect and inviting them to consider what you have in mind.

Sure, you’ll still have your rubber duck to confide in. And you can always send out for coffee. But true happiness is collaboratively debugging with a smart team that you trust. 🙂