SPDX-License-Identifier: MIT
X-SPDX-Copyright-Text: Copyright (C) 2022, Advanced Micro Devices, Inc.
Introduction
============
SolarCapture is a set of tools for capturing, timestamping, processing and
injecting network traffic. SolarCapture is able to capture packets
received from the network at very high rates, apply filtering in hardware
and software, perform custom processing, and write packets to disk in PCAP
format.
SolarCapture includes APIs for customisation and for embedding into
applications, and a flexible framework for putting together applications
from pluggable processing components.
This README file includes a brief summary of SolarCapture features.
Please see the SolarCapture User Guide for more details.
SolarCapture uses the kernel-bypass features of Solarflare network
adapters to achieve very high levels of capture and injection performance.
SolarCapture assigns highly accurate timestamps to captured packets. When
used with Solarflare's PTP-enabled adapters the capture timestamps are
synchronised across the network. On Flareon adapters timestamps can be
taken in hardware by the adapter clock, giving very precise timestamps.
SolarCapture consists of the following components:
- Command line tools for capture, replay and monitoring.
- Python bindings for creating custom configurations and applications.
- C bindings for extending SolarCapture's processing pipeline and adding
custom features.
- C bindings for embedding SolarCapture into applications.
This is the open source release of SolarCapture. Previous releases may
be downloaded from:
https://www.xilinx.com/support/download/nic-software-and-drivers.html
Dependencies
============
- A general build environment - gcc, make, libc, libc-devel and python-devel.
- libaio and libaio-devel
- libpcap and libpcap-devel
- Onload
SolarCapture uses device drivers that are included in the OpenOnload
distribution.
We advise using Onload 7 for this version of SolarCapture.
OpenOnload is available from:
https://www.xilinx.com/support/download/nic-software-and-drivers.html#open
These are needed for both building SolarCapture and running it.
If Onload is not already installed, then please note that this has more
requirements - such as kernel-devel. See the Onload User Guide for details.
SolarCapture also requires compatible hardware to run - an Onload-accelerated
network card. (e.g. SN7122F, SN8522F, SN8542Q, X2522, X2542) X3 series is
not currently supported.
SolarCapture's command line utilities require python2.
To build the html documentation you will also need doxygen. Doxygen is
available from https://www.doxygen.nl/
Installation
============
Build using "make" from the src directory.
Note that some warnings are expected, from libpcap, since SolarCapture is
using an older version which predates many of these compiler checks.
You can then run solar_capture directly.
Optionally build html documentation with "make docs"
Optionally build and run self-test with "make run"
You can then package up the libraries, examples, binaries and front end
via "make tarball". This will result in a file:
build_products/solar_capture-<VERSION>.tgz
(With appropriate version string)
This package will look much like a Linux filesystem (/usr/lib etc.)
but contains just the SolarCapture files. Before you continue - double
check that this layout matches the expectations of your Linux
distribution. (e.g. use of /lib and /lib64) It might not. If not, you
will need to manually move files to the correct places.
If it is a match, then it can be untarred to add it
to an existing Linux install via:
sudo tar -C / tar xzf build_products/solar_capture-*.tgz
Command line usage
==================
SolarCapture can be invoked from the command line. By default it
captures all traffic on an interface, which requires root privileges.
For example, to capture all traffic arriving on interface eth2 and
eth3:
$ solar_capture eth2=./eth2.pcap eth3=./eth3.pcap
To write timestamps with nanosecond resolution instead of the default
microseconds:
$ solar_capture format=pcap-ns eth2=./eth2.pcap
Note that the nano-second PCAP format is understood by a number of
tools including wireshark, but not by tcpdump.
There are a number of tunables and options for managing what streams
of packets are captured. For details run:
$ solar_capture help
To use the libpcap bindings, prefix your command line with solar_libpcap
as follows:
$ solar_libpcap tcpdump -i eth2
Monitoring
==========
The internal state of a SolarCapture process can be monitored using
the solar_capture_monitor tool.
To get a list of running SolarCapture processes:
$ solar_capture_monitor
To monitor the state of a SolarCapture process:
$ watch -d -n1 solar_capture_monitor <pid> dump
Running solar_capture_monitor has very little impact on the performance of
the capture process. Other solar_capture_monitor commands include
"line_rate" and "line_total" to show packet rate and bandwidth.
libpcap
=======
SolarCapturePro provides a modified version of libpcap, which can
capture and inject packets using the SolarCapturePro architecture,
bypassing the kernel. This can be used in two ways, either by
statically linking to the modified libpcap, or for existing
applications that dynamically link against libpcap, a wrapper script
is provided to enable the SolarCapturePro version of libpcap to be
used.
The wrapper script can be used as follows:
$ solar_libpcap <application>
Software filtering
==================
As well as filtering in hardware using the "streams=" option,
SolarCapturePro adds support for filtering in software with filters
specified in the BPF syntax. Packets that have been captured will be
matched against the specified filter, and those that do not match
will be discarded. The filter can be specified on the command line
using the "filter=<bpf-filter-string>" option. For example:
$ solar_capture eth2=/tmp/pcap filter="src host 172.16.132.99"
Software filtering is applied to all packets captured. The captured
packets can be limited by using the streams option in the normal way.
For example to capture all unicast udp traffic for a specific host
the following options could be used:
$ solar_capture eth2=/tmp/pcap streams="eth:00:0F:53:01:7D:40" filter="udp"
Sniff mode
==========
Packets captured by SolarCapture are not available by default to
the kernel stack, OpenOnload or any other process.
On Flareon adapters SolarCapturePro can be run in sniff mode
(add mode=sniff to the command line) whereby packets captured by
SolarCapture will be replicated in hardware and so also be made
available to other receivers.
The behaviour of sniff mode can be modified using the promiscuous
option. If promiscuous is enabled ("promiscuous=1" which is the
default) then all packets arriving at the sniffed interface will also
be delivered to SolarCapturePro. If promiscuous is not enabled
("promiscuous=0") then only packets that would anyway be delivered to
the host will be delivered to SolarCapturePro.
On other adapters is not possible to use SolarCapture to monitor
streams that are consumed by applications on the same server. We
recommend using SolarCapture with mirror/span switch ports.
Hardware timestamps
===================
On Flareon adapters SolarCapturePro will attempt to assign
timestamps in hardware, but fall back to software if they are not
available, e.g. due to a resource shortage or missing license.
Notes
=====
- By default, packets captured by SolarCapture are not available to the
kernel stack, OpenOnload or any other process. On Flareon adapters
you can use "mode=sniff" to capture a copy of packets arriving at the
host so that applications can continue to communicate over the same
interface.
- Timestamps are assigned either by the adapter (Flareon adapters with
SolarCapture Pro license) or by the SolarCapture software. Software
timestamps are subject to system jitter caused by the OS kernel, BIOS
and other processes. To get timestamps that are as accurate as possible,
SolarCapture should be run on isolated cores which are configured to
minimise interruptions from system interrupts and processes.
- Capture performance depends on many factors. In most deployments the
sustained capture rate is likely to be limited by storage performance.
Other factors that affect capture rate include:
* The I/O performance of the server.
* The size of the internal packet buffer pool.
* Spreading of load using receive-side scaling and application clustering.