Lib-common

The lib-common is an Intersec libc extension targeting Linux environments. It provides most of the basic tools a C programmer can expect such as:

a string manipulation library
containers like dynamic vectors, hash tables, hash maps, lists…
bitmask manipulation functions
endianness conversion primitives
…

It also contains higher level libraries:

asynchronous event loop
http client and server implementations
RPC communication channels
optimized allocators for our use cases
log file management
tracing and reporting primitives
program argument parser
ASN1 parser / writer
…

It is also the home of our IOP library. The IOP (Intersec Object Packer) is a framework to serialize structured data to use in communications protocols, data storage… IOP is language independent. IOP objects are encoded using TLV (Tag-Length-Value) packers/un-packers. The IOP concept is similar to the Google Protocol Buffers.

The IOP library is used to transmit data over the network in a safe manner. It deals with data integrity checking, retro-compatibility issues,… They are also used to exchange data between different languages or to provide a generic interface to store and load C data on disk.

Documentation

The lib-common documentation can be found in the docs directory. The documentation is based on Antora, an online version is available here.

Quickstart

This section offers basic instructions for building the lib-common.

Prerequisites

The lib-common is mainly written in C, with some extensions coming from clang features. So its main dependencies are:

glibc: used extensively, in particular to interface with various Linux kernel facilities
clang: base compiler
waf: build system
cython: to build the IOP python interface
openssl: various cryptography requirements
flex: IOP syntax lexer at https://github.com/westes/flex
gperf: GNU library for perfect hash function generation at https://www.gnu.org/software/gperf

Most of dependencies can be installed using your own Linux distribution packages, with the exception of waf which is usually not packaged.

waf installation

The lib-common build system uses waf. If you want to understand how it works, you should read the Waf Book, but this is not necessary if you just want to use it.

All the lib-common specificities and custom rules are defined in waf tools, located in the build/waftools directory.

The main entry point of the build system is the wscript file located at top-level of the repository. It loads the Intersec tools, defines the main configure and build functions, and recurses in sub-directories (that contain their own wscript or wscript_build files).

The waftools and wscript files are supposed to be readable and documented when needed.

waf can be downloaded as a single executable or built from sources, as described in the waf documentation.

The simplest approach is to download the waf executable and install it in a directory already in your PATH environment, for example /usr/local/bin:

$ cd /usr/local/bin
$ wget https://waf.io/waf-2.0.x
$ ln -s waf-2.0.x waf
$ chmod +x waf-2.0.x
$ ./waf

waf versions 2.0.8 and 2.0.18 are known to work fine, running the latest 2.0.x waf should also work fine.

Cython

To build the IOP python interface component, cython >= 0.29 is required. It can be installed using pip:

$ pip install cython

To also build the python3 IOP interface:

$ pip3 install cython

List of packages per Linux distribution

Debian 9, Debian 10 or Ubuntu 18.04

You need to install the following packages to build the lib-common:

$ sudo apt-get install python-dev python-pip flex gperf pkg-config exuberant-ctags libxml2-dev libssl-dev valgrind

To generate the python3 IOP interface, you also need the following packages:

$ sudo apt-get install python3-dev python3-pip

Fedora 31

You need to install the following packages to build the lib-common:

$ sudo dnf install clang clang-devel flex gperf libxml2-devel openssl-devel valgrind-devel python2-devel

To generate the python3 IOP interface, you also need the following package:

$ sudo dnf install python3-devel

Building lib-common

First of all, you have to configure your project, by running in the top-level directory:

$ waf configure --check-c-compiler=clang --check-cxx-compiler=clang++

Note the two options --check-xxx-compiler, in order to compile with clang, which is the easiest way to compile the lib-common. If you prefer compiling with gcc, refer to the dedicated section below.

If the configuration step triggers no error, then you are ready to build, which just consists in running:

$ waf
or
$ waf build

You can run it from a sub-directory in order to build only the targets defined in this directory and its sub-directories (and its dependencies).

All the available targets can be listed with this command:

$ waf list

It is possible to build only a specific target, or a list of targets, by running, from anywhere in the repository:

$ waf --targets=target1,target2

Other Intersec-specific waf commands

Other commands are listed with waf --help. Here they are:

waf check: run the tests of the current directory (defined in the ZFile) and in its sub-directories. The following variants also exist (cf waf --help for the details): fast-check, www-check, selenium, fast-selenium.
waf tags: generate tags using ctags.
waf etags: generate tags for emacs using ctags.
waf pylint: run pylint checks on committed python files.
old-gen-files-detect: detect old files generated by a previous build system run.
old-gen-files-delete: delete the files detected by the previous command.
coverage-start: start a coverage session (requires coverage profile). This resets the coverage counters. After running this command, you can run some code and use the coverage-end command to produce a coverage report. Note that this is done when configuring the project.
coverage-end: end a coverage session and produce a report.

Supported environment variables

The following environment variables can be used at the configuration phase:

P (string)

Specify the desired compilation profile (default, debug, release, …). The complete list of available profiles is defined in build/waftools/backend.py, variable PROFILES. If not specified, the default profile is default.

NOCHECK (boolean)

The build-system doesn’t run "check" targets, which are:

clang check of c files.
linters on js/ts files. You may want to set it to speedup the build. You can also bypass the checks thanks to the nocheck parameter of task-generators, which can be True to bypass the checks of all the source files, or a list of files to not check.

NOCOMPRESS (boolean)

If set, the build-system doesn’t compress the debug section of binaries leading to larger binary files. This can be used if you have to work with tools that does not support compressed-debug sections (like valgrind, pahole or some old version of gdb). You may also want to set it to speedup the build. This is ignored in release profile.

NO_DOUBLE_FPIC (boolean)

If set, the compilation will be faster, but the produced binaries will be larger and the runtime performances will be affected. Cf. build/waftools/backend.py for the details. This is ignored in release profile.

FAKE_VERSIONS (boolean)

If set, the version files are generated with fake (and constant) data, so that changing of git revision does not trigger a re-link of all the binaries. This is a huge gain of time, but it’s not possible to know the revision of the binaries that are built with this flag. This is ignored in release profile.

SHARED_LIBRARY_SANITIZER (boolean)

If set, the shared libraries will also be compiled with the sanitizer specified by the profile. You will have to use LD_PRELOAD or use a process that is compiled with the same sanitizer to load the compiled shared libraries with this option. This is only available for profiles that use sanitizers, i.e. asan or tsan.

Compiling with gcc

It is possible to compile the lib-common with gcc instead of clang if you prefer, but this is a bit harder.

An important part of the lib-common uses the blocks clang extension, that is not supported by gcc.

Because of this, we have put in place a two-phase build of some files (named foo.blk instead of foo.c) that are pre-compiled using a patched clang that generates gcc-compilable C code to support blocks. gcc is then used to produce the final object code.

The patched clang has to be built from this clang fork: https://github.com/Intersec/clang/tree/clang/7.0/maint and placed in your PATH environment.

Then, just configure the project without specifying the compiler (default is gcc):

$ waf configure

Contributing

In the spirit of open source software, everyone is welcome to contribute to this project!

The best way to get involved is to just show up and make yourself heard. We pride ourselves on having a very friendly and encouraging culture. Whether you’re a user, writer, designer, developer, architect, devops, system administrator, advocate, project manager, or just someone with an idea about how to improve the project, we welcome your participation. In return, you’ll get to use better software that we built together as a community.

Thanks in advance for helping to make this project a success!

Copyright and License

Licensed under the Apache License, Version 2.0 (the "License"). You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Authors

Development of the lib-common is led and sponsored by Intersec.

vthib/lib-common