Traces is an API tracing framework for Linux C/C++ applications. Is is written by Yotam Rubin <yotamrubin@gmail.com> and sponsored by infinidat. Traces is a reimplementation of similar proprietary systems implemented by Liran Zvibel, Dan Aloni and others. Complete documentation is available at <https://github.com/yotamr/traces/wiki/Home> Introduction ------------- Traces is an API tracing framework for Linux C/C++ applications. One of the main bottlenecks to development productivity is debugging. The most common debugging method is to litter one's code with trace messages in the hope that once a bug is discovered, the trace messages will be sufficient to understand its root cause. Usually this is not the case and the developer must narrow down the possible source of the bug by adding additional trace messages in the code and try to reproduce the bug again. During development, this technique can take up a lot of a developer's time. A worse scenario is having a bug appear on a client-installed application and not having sufficient trace data to understand it. Traces was written to provide rich trace information prior to the bug occurring. Traces contains a trace instrumentor, traces daemon and trace readers. Traces has the following features: * Explicit tracing through DEBUG/INFO/WARN/ERROR macros * Automatic tracing of every function leave/entry and parameters/return values * Type-aware tracing. Traces display enum value names and entire structures * Code instrumentation during compile-time * Colored trace logs * Interactive and non-interactive trace readers Traces can be used during development as well as in client-installed applications, to enable client-side debugging. Traces has a minor CPU footprint. How does it work? ----------------- Traces adds another phase of compilation, after pre-processing and before the actual compilation. During this phase, a trace instrumentor parses the code using a clang plugin. The instrumentation does the following: - Expand invocations to explicit trace points - Instrument function leave/entry points - Collect type information The clang plugin produces code that is eventually compiled by the original compiler. The instrumentation does not affect line information. To enable an application to use the traces, it must be linked against the libtraceuser.a library. traceuser allocates the shared memory area the records are written to. A separate process trace_dumper attaches to the shared memory regions of each traced process and dumps the records to a journal file as well as metadata necessary to parse the records. Quick start ----------- traces has the following dependencies: clang 3.1 Python 2.7 or above python-ctypeslib (for the interactive reader) python-urwid (for the interactive reader) gccxml Note: if python-ctypeslib is not packaged for your distro, then install python-pip instead and try one of these: # for fedora based distros sudo pip-python install ctypeslib # for debian based distros sudo pip install ctypeslib To build the traces framework, change directory to the root of the traces dir and run `make all`. To compile traces with existing code: Replace compiler invocations to '$PATH_TO_CCWRAP $CC', where PATH_TO_CCWRAP is the absolute path of the ccwrap.py script Link the final executable with the libtraceuser.a library Run the traced executable After your application has been compiled with traces enabled, you can quickly start examining its traces: Run trace_dumper -o (to produce verbose output showing function entry/leave, use the '-d' parameter. trace_dumper will show trace messages as they are generated by the traced application. Run the traced application. The trace_dumper should start printing traces immediately. Normally, trace_dumper is used to write records directly to file rather than parse and display them online. To run trace_dumper in write mode, invoke it with the -w[filename] argument, optionally passing a destination filename. If no filename is specified, trace files are written to /mnt/logs and are rotated once the trace file reaches 1 GB. Reading traces -------------- There are two ways to view the contents of a trace file: simple_trace_reader and the interactive reader. non-interactive reader: trace_reader/simple_trace_reader -d $filename interactive-reader: python interactive_reader/ui.py $filename The non-interactive displays only non-function and non-debug records. This behavior can be controlled through the command line. The interactive reader supports searching and filtering according to the following criterias. See the help (press 'h' in the main display) for exact usage examples: * severity * Function name * Function parameter value and name * Function return value * Struct field value and name * Search by tid, pid, process name, * Search by type name referenced in trace record * Search by time and time range * Boolean operations (not, or, and)