pflua

pflua is a high-performance network packet filtering library written in Lua. It supports filters written in pflang, the filter language of the popular tcpdump tool. It's really fast: to our knowledge, it's the fastest pflang implementation out there, by a wide margin. Read on for more details.

Getting started

$ git clone --recursive https://github.com/Igalia/pflua.git
$ cd pflua; make             # Builds embedded LuaJIT
$ make check                 # Run builtin basic tests

Using pflua

Pflua is a library; you need an application to drive it.

The most simple way to use pflua is filtering packets from a file captured by tcpdump. For example:

$ cd tools
$ ../deps/luajit/usr/local/bin/luajit pflua-filter \
    ../src/ts/pcaps/ws/v4.pcap /tmp/foo.pcap "ip"
Filtered 43/43 packets from ../src/ts/pcaps/ws/v4.pcap to /tmp/foo.pcap.

See the source of pflua-filter for more information.

Pflua was made to be integrated into the Snabb Switch user-space networking toolkit, also written in Lua. A common deployment environment for Snabb is within the host virtual machine of a virtualized server, with Snabb having CPU affinity and complete control over a high-performance 10Gbit NIC, which it then routes to guest VMs. The administrator of such an environment might want to apply filters on the kinds of traffic passing into and out of the guests. To this end, we plan on integrating pflua into Snabb so as to provide a pleasant, expressive, high-performance filtering facility.

Given its high performance, it is also reasonable to deploy pflua on gateway routers and load-balancers, within virtualized networking appliances.

Implementation

Pflua can compile pflang filters in two ways.

The default compilation pipeline is pure Lua. First, a custom parser produces a high-level AST of a pflang filter expression. This AST is lowered to a primitive AST, with a limited set of operators and ways in which they can be combined. This representation is then exhaustively optimized, folding constants and tests, inferring ranges of expressions and packet offset values, hoisting assertions that post-dominate success continuations, etc. Finally, we residualize Lua source code, performing common subexpression elimination as we go.

The resulting Lua function is a predicate of two parameters: the packet as a uint8_t* pointer, and its length. If the predicate is called enough times, LuaJIT will kick in and optimize traces that run through the function. Pleasantly, this results in machine code whose structure reflects the actual packets that the filter sees, as branches that are never taken are not residualized at all.

The other compilation pipeline starts with bytecode for the Berkeley packet filter VM. Pflua can load up the libpcap library and use it to compile a pflang expression to BPF. In any case, whether you start from raw BPF or from a pflang expression, the BPF is compiled directly to Lua source code, which LuaJIT can gnaw on as it pleases.

We like the independence and optimization capabilities afforded by the native pflang pipeline. However, though pflua does a good job in implementing pflang, it is inevitable that there may be bugs or differences of implementation relative to what libpcap does. For that reason, the libpcap-to-bytecode pipeline can be a useful alternative in some cases.

See the doc subdirectory for some examples of the Lua code generated for some simple pflang filters using these two pipelines.

Performance

To our knowledge, pflua is the fastest implementation of pflang out there. See https://github.com/Igalia/pflua-bench for our benchmarking experiments and results.

Pflua can beat other implementations because:

LuaJIT trace compilation results in machine code that reflects the actual traffic that your application sees
Pflua can hoist and eliminate bounds checks, whereas BPF is obligated to check that every packet access is valid
Pflua can work on data in network byte order, whereas BPF must convert to host byte order
Pflua takes advantage of LuaJIT's register allocator and excellent optimizing compiler, whereas e.g. the Linux kernel JIT has a limited optimizer

API documentation

None yet. See pf.lua for the high-level compile_filter interface.

Bugs

Check our issue tracker for known bugs, and please file a bug if you find one. Cheers :)

Authors

Pflua was written by Andy Wingo, Diego Pino, and Javier Muñoz at Igalia, S.L.. Development of pflua was supported by Luke Gorrie at Snabb Gmbh, purveyors of fine networking solutions. Thanks, Snabb!

Feedback is very welcome! If you are interested in pflua in a Snabb context, probably the best thing is to post a message to the snabb-devel group. Or, if you like, you can contact Andy directly at wingo@igalia.com. If you have a problem that pflua can help solve, let us know!

liveck/pflua