Installing Java Agents to Existing Kubernetes and Helm Applications Instantly

In a recent blog post, one of my teammates, Josh, shared a few techniques for deploying Java agents in Kubernetes applications. We have been getting a lot of interest in the concepts we have shared and, per popular request, decided to raise the bar. Is it possible to add a Java agent without changing a single line in either the Dockerfile or the Kubernetes Manifest? Well, the answer is most definitely yes (!), and here’s how.

In case you are not familiar with them, Java agents are jar packages used to instrument a Java application. Java agents empower tools such as Rookout and APMs to provide insights into what’s going on in a running application. You add a Java agent in an application using the JVM flag -javaagent (read more about it here).

Quick Recap

In that previous blog post, Josh demonstrated how to edit an existing Kubernetes application manifest to add a Java agent.

The first step is to add an init container that will add the Java agent jar file to the pod’s filesystem. The second step is to add the JAVA_TOOL_OPTIONS environment variable with the -javaagent flag, instructing the JVM to load it. The final manifest will look a bit like this:

apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
labels:
app: myapp
spec:
containers:
- name: java-app-container
image: <image-name>
env:
- name: JAVA_TOOL_OPTIONS
value: -javaagent:/shared/java-agent/rook.jar
initContainers:
- name: init-agent

‍

The final step is to create the container image for the init container that will add the Java agent to the pod’s file system. I have created a Github repository for your convenience and made the container image available on Docker Hub.

How can we do it without changing the manifest?

‍

Using the technique mentioned above, all we have to do to install the Java agent is to add a few configuration options to the Kubernetes manifest. Unfortunately, that means we need to either change or fork the upstream manifest file.

For some of our customers, neither of those choices is appropriate, and that’s where Kustomize comes in. If you are not familiar with it, Kustomize adds, removes, or modifies Kubernetes manifests on the fly. It is even available as a flag to the Kubernetes CLI – simply execute kubectl -k.

For this example, we’ll be using Arun’s open-source Hello World Java application. For simplicity, I have taken the Kubernetes manifest, upgraded the APIs for the latest Kubernetes version, and made it available here.

We start by creating a kustomization.yaml file, the root of Kustomize based deployments. The file is relatively short and straightforward and comprises a reference to the application manifest, the patch to apply to insert the Java agent, and a secret generator for the token. Here it is:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml # This is the original application we are deploying
patchesStrategicMerge:
- java-agent.yaml # Here, we add the Java agent
secretGenerator:
- name: rookout
literals:
- token=<ROOKOUT_TOKEN> # Create any secrets the Java agent needs

Now we have to write the Kustomize patch itself. The patch has to take care of the following tasks:

1. Mount a shared volume in the application pod.
2. Set the needed environment variables in the application pod.
3. Add the init container that will place the Java agent jar in the shared volume.
4. Create the shared volume itself.

Here’s the final patch:

apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world
spec:
template:
spec:
containers:
- name: hello-world-pod # Edit the main application container
volumeMounts:
- mountPath: /shared/java-agent # Mount the directory where the Java agent will be available
name: java-agent
env:
- name: JAVA_OPTIONS # Instruct the JVM to load the Java agent
value: -javaagent:/shared/java-agent/rook.jar
- name: ROOKOUT_TOKEN # Make the token available as an environment variable
valueFrom:
secretKeyRef:
name: rookout
key: token
initContainers: # Add an init container that will drop the Java agent to the shared volume
- name: init-agent
image: rookout/add-java-agent:latest
volumeMounts:
- mountPath: /shared/java-agent
name: java-agent
volumes: # Create the shared volume
- name: java-agent
emptyDir: {}

‍

To see it in action, all you have to do is:

1. Clone the repository git clone https://github.com/Rookout/k8s-java-agent
2. Set the Rookout token in the kustomization.yaml file
3. Deploy the application by running kubectl apply -k kustomize

What about Helm?

‍

At this point, you might be wondering, that’s a powerful approach, but my team does not use kubectl apply. We are using Helm, the Kubernetes package manager, to define, publish, and install our applications. And in the past, using Kustomize in addition to Helm has been quite cumbersome.

If that’s the case, I have some fantastic news for you. The release of Helm v3.1 in early 2020 changed that. Helm’s post rendering feature allows us to run Kustomize on a fully rendered Helm chart just before deploying it.

For this example, I’ll again use Arun’s Hello World application. He has kindly built a Helm chart, which I have updated for the latest version of Kubernetes and placed here. To migrate our existing Kustomize configuration to work with Helm, we have to do just a handful of steps.

First, we have to install Kustomize as a standalone application and place it in the PATH. Second, we need to mediate between Helm, which writes to the standard output, and Kustomize, which expects to read everything from a file. Finally, we need to create our Kustomize configuration. As it turns out, we can use the same configuration, so we just copy it to our working directory.

The final script looks like this:

# !/bin/bash# save incoming YAML from Helm to file
cat <&0> kustomize/deployment.yaml
# Copy our original Kustomize config
cp ../kustomize/kustomization.yaml kustomize/
cp ../kustomize/java-agent.yaml kustomize/
# modify the YAML with kustomize and print it to the standard output
kustomize build kustomize
rm kustomize/*.yaml

‍

Now, all we have to do to install the Helm chart with our Java agent is to run:

helm install hello-world docker-kubernetes-hello-world --post-renderer kustomize/kustomize

‍

Summary

‍

Installing a Java agent is one of the most basic tasks when developing and operating Java applications, and moving to the cloud-native ecosystem doesn’t make it any less significant. Installing those agents by changing the container image is a relatively simple process but has multiple drawbacks. For many of our customers, installing Java agents has traditionally been the operations team’s responsibility, and that’s where it should remain.

In this blog post, I have shared an alternative approach to installing Java agents in existing applications, at deployment time, without requiring any upstream changes. I hope you’ll not only find it useful but benefit from a deeper understanding of a few of the building blocks I have shared with you.