/golang-bpf

Work-in-progress guidelines for using bpftrace with go applications

Primary LanguageShell

BPF for Go

A work-in-progress repository of guidelines for using bpftrace with go applications

Project Goals

  • document a reusable workaround for bpftrace uretprobes
  • produce a collection of "snippets" for using bpftrace w/ golang applications
  • document the process by which I worked out all the workarounds and snippet described above
  • demonstrate usage of bpftrace on Nomad, which is the project I'm working on full-time
  • produce some kind of e-book to share this is a nicer format than this messy repo

Non-Goals

  • dealing with arbitrary BPF programs beyond bpftrace (i.e. no bcc-tool, gobpf, etc.)

Progress

Examples:

  • Hello world
  • Minimal stack arguments tracing (minimal)
  • Single integer return tracing (minimal)
  • Multiple return tracing (minimal)
  • Function latency example (worker, web)
  • Working with goroutines (worker, web)
  • Unpacking structs and slices from results (worker, web)
  • Mixing application-level and OS-level tracing (Nomad)

Documentation:

  • Introduce BPF and bpftrace (probably borrow from my DevOps Data Philly workshop)
  • Explain why golang is weird and special for BPF vs C
  • Very light introduction to reading assembly of golang programs
  • Curated pointers to better reference material

Setup

Tool Install

Use Vagrant with the Vagrantfile at the root of this repo.

  • run vagrant up to start the machine and provision it
  • starts with bento/ubuntu-19.10 machine
  • adds scripts from Nomad for golang development
  • add scripts for installing bpftrace and tools
  • run vagrant ssh to shell in

Repo Contents

  • Vagrantfile is the definition of the virtual machine used to run the experiments.
  • GNUmakefile is the makefile for building the workloads used in the experiments.

Directories in this repo:

  • ./notes contains notes of my exploration of using bpftrace to examine golang applications. They include plenty of false starts and are generally rough-draft quality.
  • ./provisioning contains shell scripts used by the Vagrantfile at the root of the repo to stand up an Ubuntu box with a recent Linux kernel and the tools needed to run through the exercises in the notes.
  • ./scripts contains some example bpftrace scripts used by the examples in the notes.
  • ./targets contains some example workloads to trace.

Directories created by this repo:

  • ./.vagrant is the state of the Vagrant-managed virtual machine.
  • ./bin is the binaries of the example workloads that the makefile will build.
  • ./bpftrace is the source repo for bpftrace.

Target Workloads

  1. a hello world to repro Brendan Gregg's blog but w/ bpftrace
  2. a minimal application that makes some function calls
  3. a worker application that has concurrency (goroutines), defers, multiple returns, and slightly more complex parameter and results parsing.
  4. [TBD] a web application that does JSON parsing (reflection), calls into sqlite (CGO)
  5. Nomad

References