Java Cloud Debugger Agent

Overview

Cloud Debugger (also known as Stackdriver Debugger) lets you inspect the state of a running cloud application, at any code location, without stopping or slowing it down. It is not your traditional process debugger but rather an always on, whole app debugger taking snapshots from any instance of the app.

Cloud Debugger is safe for use with production apps or during development. The Java debugger agent adds less than 10ms to the request latency when a debug snapshot is captured. In most cases, this is not noticeable to users. Furthermore, the Java debugger agent does not allow modification of application state in any way, and has close to zero impact on the app instances.

Cloud Debugger attaches to all instances of the app providing the ability to take debug snapshots and add logpoints. A snapshot captures the call-stack and variables from any one instance that executes the snapshot location. A logpoint writes a formatted message to the application log whenever any instance of the app executes the logpoint location.

The Java debugger agent is only supported on Linux at the moment. It was tested on Debian Linux, but it should work on other distributions as well.

Cloud Debugger consists of 3 primary components:

The Java debugger agent (requires Java 7 and above).
Cloud Debugger service storing and managing snapshots/logpoints. Explore the API's using APIs Explorer.
User interface, including a command line interface gcloud debug and a Web interface on Google Cloud Console. See the online help on how to use Google Cloud Console Debug page.

Getting Help

StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-debugger
Send email to: Cloud Debugger Feedback
Send Feedback from Google Cloud Console

Installation

The easiest way to install the Java debugger agent for Google Cloud Platform is to download the pre-built package from the Internet. (the package is updated periodically):

mkdir /opt/cdbg
wget -qO- https://storage.googleapis.com/cloud-debugger/compute-java/debian-wheezy/cdbg_java_agent_gce.tar.gz | \
    tar xvz -C /opt/cdbg

Alternatively you can build the Java debugger agent from source code:

git clone https://github.com/GoogleCloudPlatform/cloud-debug-java.git
cd cloud-debug-java
chmod +x build.sh
./build.sh
ls cdbg_java_agent_gce.tar.gz

Note that the build script assumes some dependencies. To install these dependencies on Debian, run this command:

sudo apt-get -y -q --no-install-recommends install \
    curl gcc build-essential libssl-dev unzip openjdk-7-jdk \
    cmake python maven

Setup

The Java debugger agent is a JVMTI agent that needs to be enabled when JVM starts with the -agentpath option of the Java launcher. Most of the debugger options are configured through system properties.

For example:

java -agentpath:/opt/cdbg/cdbg_java_agent.so -jar ~/myapp.jar

By default the Java debugger agent assumes that it runs on Google Cloud Platform and obtain the credentials from the local metadata service. To use the Java debugger agent outside Google Cloud Platform requires setting up a service account.

Application Servers

Java application servers usually start through a bootstrap process, and each application server has its own way of customizing Java options.

Tomcat

Add this line to /etc/default/tomcat7 or /etc/default/tomcat8:

JAVA_OPTS="${JAVA_OPTS} -agentpath:/opt/cdbg/cdbg_java_agent.so"

If you run Tomcat in a Docker container, add this line to Dockerfile instead:

ENV JAVA_OPTS -agentpath:/opt/cdbg/cdbg_java_agent.so

Jetty

Add cdbg.ini file to /var/lib/jetty/start.d:

--exec
-agentpath:/opt/cdbg/cdbg_java_agent.so

Extra Classpath

The Java debugger agent needs to be able to find the application classes when it's running in an application server like Tomcat or Jetty. By default, it looks for the exploded root war directory. In other words, if you deployed a ROOT.war in Tomcat, the agent can find it without additional configuration.

However, if you deployed your WAR file with a different name (e.g., myapp.war), or that the exploded WAR directory is not under the default exploded root war directory (e.g., your exploded war is under /opt/tomcat/webapps/myapp), then you must let the agent know the full path to your application's classes using the cdbg_extra_class_path parameter.

-agentpath:/opt/cdbg/cdbg_java_agent.so=--cdbg_extra_class_path=/opt/tomcat/webapps/myapp/WEB-INF/classes

You can specify multiple paths by using a : (colon) as the path delimiter.

-agentpath:/opt/cdbg/cdbg_java_agent.so=--cdbg_extra_class_path=/opt/tomcat/webapps/myapp/WEB-INF/classes:/another/path/with/classes

Naming and Versioning

Developers can run multiple applications and versions at the same time within the same Google Cloud Platform project. You should tag each app version with the Cloud Debugger to uniquely identify it in the Cloud Debugger user interface.

To tag the application and it's version, please add these system properties:

-Dcom.google.cdbg.module=my-app-name
-Dcom.google.cdbg.version=my-app-version

Use module to name your application (or service). Use version to name the app version (e.g. build version). The UI will display the running version as module - version.

Logging

By default the Java debugger aget writes its logs to cdbg_java_agent.INFO file in the default logging directory. It is possible to change the log directory as following:

-agentpath:/opt/cdbg/cdbg_java_agent.so=--log_dir=/my/log/dir

Alternatively you can make the Java Cloud Debugger log to stderr:

-agentpath:/opt/cdbg/cdbg_java_agent.so=--logtostderr=1

Service Account

To use the Java debugger agent on machines not hosted by Google Cloud Platform, the agent must use a Google Cloud Platform service-account credentials to authenticate with the Cloud Debugger Service.

Use the Google Cloud Console Service Accounts page to create a credentials file for an existing or new service-account. The service-account must have at least the Cloud Debugger Agent role to be accepted by the Cloud Debugger Service. If you don't have a Google Cloud Platform project, you can create one for free on Google Cloud Console.

Once you have the service-account JSON file, deploy it alongside the Java debugger agent.

Using the service-account option requires the Java debugger agent version that supports it. Either download the pre-packaged agent from https://storage.googleapis.com/cloud-debugger/compute-java/debian-wheezy/cdbg_java_agent_service_account.tar.gz or the locally built cdbg_java_agent_service_account.tar.gz

To use the service-account credentials add these system properties:

-Dcom.google.cdbg.auth.serviceaccount.enable=true
-Dcom.google.cdbg.auth.serviceaccount.jsonfile=/opt/cdbg/gcp-svc.json

Alternatively, you can set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the JSON file path instead of adding the auth.serviceaccount.jsonfile system property.

Other JVM Languages

Starting with version 2.13, you can use the Java Cloud Debugger to debug Scala applications. You can take snapshots in both .scala and .java source files, however snapshot expressions and conditions must always be written using the Java programming language syntax.

quoctruong/cloud-debug-java