Logging Best Practices: What You Should Be Looking For

Debug logs are incredibly valuable for the wealth of information they contain, but generally speaking, companies avoid collecting them in production because they are very expensive in both dollars and performance.

When we talk about production observability, we are likely spending more money for more data, data that itself might be excessive. The highest verbosity-level logs, TRACE and DEBUG, bring with them valuable data points that can help locate that needle in a haystack, but they could just as easily be heaping more hay onto the pile. 

These are among the several challenges in production logging.Some of those issues are similar to debugging, but some can be pretty unique to logs.

Common Production Logging Issues & Trade-Offs

• LOW-INFO, HIGH-NOISE – Most DevOps teams will limit the types of logs they keep at INFO, WARNING, or ERROR levels. These are the least verbose, and consequently the least informative, logs you can have. Despite what data they do give, they’re not specific enough and can end up just being clutter or noise. If they don’t seem worth checking, then devs might start ignoring all errors, even missing out on relevant logs having relegated them to that clutter. In other words, too many irrelevant logs will get devs to ignore logs entirely.

• HIGH STORAGE-HIGH COST – By the same token, these production logs end up being stored somewhere. Even if kept for a limited amount of time, they are draining your coffers because of storage costs and performance impact.

• HIGH RISK, LOW REWARD, EVEN HIGHER COSTS – It can be risky implementing these kinds of changes in a live environment, probably outweighing the likely reward of going through the trouble to implement them. And of course, there’s the expense – mo’ data means mo’ money and mo’ problems for your wallet.

• SLOWED AND STOPPED APPS  – Best practice in this situation would be to fully stop the application and restart it with new log settings, which is a major hit for any organization. Even with that change, controlling the amount of DEBUG and TRACE logs can become a chore in and of itself. 

• LOTS OF TALK, LITTLE WALK – Even if those four aforementioned forks in the workflow weren’t issues, you now have a deluge of extra log lines to deal with in parsing and storage.

​​Instead, you need to understand the best practices for live logging. Below are some of the main points in more detail.

Dynamic Log Verbosity

Log verbosity is a double-edged sword. Clearly, more data means more chances to discover something important. But it can just as easily be a lot of noise. 

Let’s take the developer-first approach for a second: Is this making your dev team’s job easier? If not, what would? Make it simple to switch between log levels.

Everyone on your team – with a single click and without relying on anyone else’s permissions – should be able to change log levels. They should be able to quickly shift from DEBUG level to TRACE and back again, or adjust it to INFO or ALERT, when the situation calls for it. Notice the need, implement it, and that’s it. There’s no need to bury a reconfiguration into a to-do list. This is a more dynamic, developer-friendly setup for logging.

Observability solutions should take pains to reduce that noise by 1) highlighting certain logs over others and 2) applying context to consider which logs to save and which to get rid of. The option to further filter those results on a conditional basis – e.g., don’t log geolocation on Sundays – goes hand in hand with that.

Contextual Filtering

What makes the biggest difference is applying logic to filter what information you actually need from within your logs. Contextual filtering would create a set of conditions for highlighting certain kinds data and trimming others. From the get-go, this cuts down on a lot of the noise.

Additionally, you should have more granular control over logs. When identifying user data, drill-downs should take you to certain sets of, or specific individual, accounts and users.

All developers can view the logs they want in a controlled environment without teammates having to see them in the main account. 

Finally, verbosity should be adjustable by other conditions such as specific services, selected functions, or designated files. Of course, these settings should be dynamic – you won’t know immediately what settings to impose on each function, service, or file throughout your source code, at least not immediately. As issues present themselves, you can adapt those settings and increase logging precision.

Log Aggregation

Integrations might go without saying, but in a crowded marketplace, this is one way to separate the grain from the chaff. Use cases for tools change often, so while you might not be keen to make frequent changes to your tech stacks, you want the flexibility to mix and match different options. 

If you have a certain tool designated for individual logging tasks, and yet another for storing those logs, you’ll need maximum flexibility and as many options as possible for exporting those logs.

Enriching Your Data 

Live logging should complement your existing tools. An ideal live logger would be flexible enough to produce rich logs for all log levels – DEBUG, TRACE, ALERT, and the rest. These more verbose logs need to be properly used in the correct context and honed in on the relevant data without the torrential downpour of unnecessary information, and then switched off when you don’t need them.

Efficient Live Logging in Production

There are a lot of tools out there for observability, but few can properly juggle all these above considerations. Rookout’s Live Logger does this, built with the same philosophy as our Live Debugger to maximize productivity – with a click of a button you can crank up your log verbosity on verbose logs, get the data that you need, then switch them back off immediately after. TRACE-level and DEBUG-level logs are incredibly valuable weapons in the Dev and DevOps arsenal. 

It is a post-modern tragedy when teams can’t use them in production. The Live Logger changes that with contextual filtering and dynamic log verbosity. Finally, you retain your existing logging tools for storage and organization through our integrations with tools like Dynatrace, Datadog, Sumo Logic, and more. Give the Live Logger a spin.