unable to install lttng on rt linux kernel
flynneva opened this issue ยท 9 comments
so I recently ran into this problem while trying to setup ros2_tracing
to track system performance. essentially I was unable to sudo apt install lttng-modules-dkms
as it said the RT kernel was not officially supported.
I did some digging online and came across this thread that looks to be exactly the same issue.
It goes on to suggest a solution of compiling lttng
from source....which ideally i would like to avoid.
As lttng
is a pretty universal tracer tool I was wondering if an alternative proposal could happen where we add it to the Dockerfile
here so that we can compile it right into the kernel for use later on.
I found this README
for lttng-modules
that looks like it has instructions to add it directly to a kernel build.
### Kernel built-in support
It is also possible to build these modules as part of a kernel image. Simply
run the [`scripts/built-in.sh`](scripts/built-in.sh) script with the path to
your kernel source directory as an argument. It will symlink the
lttng-modules directory in the kernel sources and add an include in the kernel
Makefile.
Then configure your kernel as usual and enable the `CONFIG_LTTNG` option.
### Required kernel config options
Make sure your target kernel has the following config options enabled:
- `CONFIG_MODULES`: loadable module support (not strictly required
when built into the kernel),
- `CONFIG_KALLSYMS`: see files in [`wrapper`](wrapper); this is
necessary until the few required missing symbols are exported to GPL
modules from mainline,
- `CONFIG_HIGH_RES_TIMERS`: needed for LTTng 2.x clock source,
- `CONFIG_TRACEPOINTS`: kernel tracepoint instrumentation
(enabled as a side-effect of any of the perf/ftrace/blktrace
instrumentation features).
- `CONFIG_KPROBES` (5.7+): use kallsyms for kernel 5.7 and newer.
would adding lttng
to the kernel build process be an acceptable enhancement here? or is it not acceptable to add it to the default kernel build?
@flynneva yes, we should do it. It was not clear to me whether RT
and lttng
work together, but apparently they do.
Do you want to give it a try and provide a PR?
I was planning on trying do this soon, but if you can figure out how to do it it'd be great!
so I think I got this working but I wont know for sure until the build fully completes.
you can see what I did in #18 in the Dockerfile
to get it added to the kernel. the trickiest bit was figuring out that we also needed the scripts/rt-patch-version.sh
which I only guessed at since it wasnt anywhere in the docs that I could find.
what makes me think I have it right is during the build, right at the beginning it prompted me a few questions specifically about LTTng and its configuration...so fingers crossed.
would love to have someone more familiar with lttng
and ros2_tracing
check my work over on #18, however I think I've got it working following the normal kernel build instructions from the README
.
the only change to the process that i did was i skipped sudo apt install lttng-dkms-modules
since we built it into the kernel. everything seems to be working OK...however I dont fully understand if what im seeing in the tracing metadata is "right" or not.
maybe @christophebedard @razr or @LanderU knows more and can understand if things look OK?
I think it looks good! The metadata file always looks a bit weird (it's not 100% text I think?). You can check the CTF spec if you want (https://diamon.org/ctf/), but it's probably not worth it ๐
You could simply take a look at the userspace and kernel traces separately to make sure stuff was recorded:
$ babeltrace /home/ubuntu/.ros/tracing/my-tracing-session/kernel
$ babeltrace /home/ubuntu/.ros/tracing/my-tracing-session/ust
With the default setup, the kernel trace will probably have a lot more events, which you can count by piping the babeltrace output to wc -l
.
awesome to hear @christophebedard, thanks! yes I removed a bunch of gibberish at the beginning (non-text stuff) when copying to github so thats probably what you were talking about.
looking good:
babeltrace /home/ubuntu/.ros/tracing/my-tracing-session/kernel | wc -l
63546
babeltrace /home/ubuntu/.ros/tracing/my-tracing-session/ust | wc -l
130
thanks for the tips!