ReMon

Introduction

This repository hosts ReMon, a secure and efficient Multi-Variant Execution Environment for x86 Linux programs. ReMon consists of two main components:

The GHent University Multi-Variant Execution Environment (GHUMVEE). GHUMVEE is a full-fledged ptrace-based MVEE. It supports over 200 system calls and it is capable of running a wide variety of realistic programs.
The In-Process MONitor (IP-MON). IP-MON is a highly efficient user-space monitor that can replicate ~70 distinct system calls without reporting them to GHUMVEE. Contrary to GHUMVEE, IP-MON is not a full MVEE. It can merely be used to augment GHUMVEE and to enhance its performance.

GHUMVEE supports both the AMD64 and the i386 architectures, though the latter has not been maintained for quite some time. IP-MON supports only the AMD64 architecture.

The current version of IP-MON takes quite a lot of manual effort to set up. Unless you REALLY need maximum performance, I would recommend using only GHUMVEE, and not IP-MON.

ReMon Prerequisites

You will need:

A GNU/Linux distribution based on Debian. I strongly recommend Ubuntu 18.04 x64.
Ruby
CMake (>= 3.4.3)
The ReMon toolchain, which can be installed using the bootstrap.sh script.

GHUMVEE Instructions

Building GHUMVEE

Building GHUMVEE is really easy. Just navigate to ReMon's root folder and type make -f makefile.release. This will build an optimized and statically linked version of the GHUMVEE binary.

Alternatively, you can also use these build configurations:

Release-syms: link-time optimized version of GHUMVEE (with symbol tables intact). Suitable for people who want to debug the Release build for some obscure reason. Build using make -f makefile.release-with-syms.
Debug: unoptimized version of GHUMVEE. This builds really fast and is suitable for people who want to debug GHUMVEE. Build using make -f makefile.debug.
Debug-sanitize: unoptimized version of GHUMVEE with address-sanitizer enabled. Might be useful to debug memory corruption bugs. Build using make -f make.debug-sanitize

You will find the compiled GHUMVEE binary in the MVEE/bin// folder.

Configuring GHUMVEE

GHUMVEE contains a number of configurable options and features. Features that severely impact GHUMVEE's performance generally must be configured at compile time by editing the /path/to/ReMon/MVEE/Inc/MVEE_build_config.h file. Don't forget to recompile GHUMVEE after editing this file.

Features/options with minimal performance impact can be configured by editing the MVEE.ini file in the output folder for your selected GHUMVEE build type (i.e. /path/to/ReMon/MVEE/bin/Release/ or /path/to/ReMon/MVEE/bin/Debug/).

The main feature that you might want to use is debug logging. Debugging logging can be enabled by disabling the MVEE_BENCHMARK feature in /path/to/ReMon/MVEE/Inc/MVEE_build_config.h.

Configuring the kernel (AMD64 systems only)

On x64, the vsyscall and vDSO need to be disabled or hidden because they contain a shared read-only page with timing info. The kernel periodically updates this timing information so that user-space programs do not neccessarily have to go into the kernel to perform calls like sys_gettimeofday. From an MVEE's perspective however, they are a problematic source of inconsistencies (we cannot easily intercept reads from the vDSO/vsyscall pages so different variants might read different timing values and might diverge because of it).

GHUMVEE contains a loader-tool that is transparently injected and that will hide the vDSO. However, the vsyscall page can only be hidden if your kernel is configured to use native vsyscall pages (this used to be the default setting, but it no longer is). To enable native vsyscall pages, use the following commands:

sudo vi /etc/default/grub
# find the line that says GRUB_CMDLINE_LINUX_DEFAULT and add vsyscall=native to it
sudo update-grub
sudo reboot

Running GHUMVEE

Depending on which build type you selected, you'll find GHUMVEE in either /path/to/ReMon/MVEE/bin/Release/ or /path/to/ReMon/MVEE/bin/Debug/. Navigate to this folder and you'll find the MVEE executable.

You can launch the MVEE in two modes:

Legacy Mode: Use ./MVEE [Builtin Configuration Number (see MVEE_config.cpp)] [Number of Variants] [MVEE Options].
RAVEN Mode: Use ./MVEE -s [Variant Set (default: default)] -f [Config File (default: MVEE.ini)] [MVEE Options] -- [Additional Program Args]

To see a full list of supported option, just launch the MVEE using ./MVEE. The config file format is (mostly) compatible with RAVEN. A full overview of RAVEN's options is available in RAVEN-config.pdf.

NOTES:

The -- is mandatory in RAVEN mode, even if no program args are passed.
The default config file is set up to launch two variants of /bin/bash -c by default. Thus, if you were to launch ReMon like this: ./MVEE -- "echo test", this would end up executing /bin/bash -c echo test.

Shutting GHUMVEE down

The easiest way to shut GHUMVEE down is to use CTRL+C. This will kill all of the variants and safely terminate the MVEE.

Alternatively, you might want to use the MVEE_backtrace tool in /path/to/ReMon/Utilities/MVEE_backtrace/ to shut the MVEE down. As the name suggests, the MVEE_backtrace tool will force GHUMVEE to generate stack traces for all of the variants it's currently monitoring. Keep in mind, however, that MVEE_backtrace will only work if you compile GHUMVEE with debug logging enabled.

IP-MON Instructions

To fully unleash the power of ReMon, you will also need to compile and enable IP-MON. Do keep in mind, however, that IP-MON is not nearly as mature as GHUMVEE. It should work, but it takes quite some effort to get it up and running, and there might still be bugs.

If you're not looking to maximize performance, then please skip this entire section. Debugging divergences is MUCH easier if you just use GHUMVEE without IP-MON.

Building IP-MON

To build IP-MON itself, navigate to /path/to/ReMon/IP-MON and type ./comp.sh.

Building the IP-MON kernel

IP-MON requires some kernel modifications to run. ReMon ships with the necessary kernel patch for Linux 4.40. To build and install the custom kernel, use the following commands:

cd /wherever/you/want/to/download/the/kernel
git clone -b linux-4.4.y git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
cd linux-<insert version number here>
patch -p1 < /path/to/ReMon/patches/linux-4.4.0-full-ipmon.patch
make menuconfig 
# while you're in the config menu, you might want to bump the kernel tick rate up to 1000Hz
# you can do so by navigating to "Processor type and features" > "Timer Frequency"
make -j 8
sudo make modules_install
sudo make install

Configuring the IP-MON policy

The IP-MON syscall policy must be selected at the source level. You can do so by editing the /path/to/ReMon/IP-MON/MVEE_ipmon.h file. Here, you can edit the definition of CURRENT_POLICY to select your policy.

Enabling IP-MON

To enable IP-MON, simply edit the MVEE.ini file in the output folder of your selected GHUMVEE build type and set the use_ipmon option to 1.

Running IP-MON

You cannot run IP-MON directly. GHUMVEE will automatically load and run IP-MON when you enable it using the use_ipmon option described above.

Older IP-MON version

Although we no longer support the version of IP-MON we presented at USENIX ATC, you can still find its source code in this repository in the IP-MON-atc folder.

Further Tinkering

Building GHUMVEE-ready glibc and libpthreads libraries

To run multi-threaded variants, we need a glibc and libpthreads that can replicate any synchronization decisions made by the master variant to the slave variants. You can read all about this in my paper called "Multi-Variant Execution of Parallel Programs".

GHUMVEE ships with prebuilt versions of glibc and libpthreads. These prebuilt versions are loaded into the variants' address spaces automagically. If you're not planning to do anything crazy, these prebuilt versions will suffice and you can skip the rest of this section.

Some people might want to build their own GHUMVEE-ready versions of glibc and libpthreads. They can do so as follows:

# get the source code for GHUMVEE's glibc
git clone https://github.com/stijn-volckaert/ReMon-glibc.git

# build
cd ReMon-glibc
mkdir build-tree
cd build-tree

# set up the makefiles. Alternatively, you can run ../configure-libc-partial-order-debug.sh here
# to compile glibc with a synchronization agent that has self-debugging features
../configure-libc-woc.sh
make -j 8

# install the libraries into $HOME/glibc-build
make install

The current version of ReMon will load the glibc and libpthreads in the /path/to/ReMon/MVEE/patched_binaries/libc// folder into each of the variants' address spaces. You should set up symlinks in this folder so ReMon loads the IP-MON-compatible glibc/libpthreads.

Building GHUMVEE-ready versions of libstdc++ and libgomp

In addition to GHUMVEE-ready versions of glibc and libpthreads, some multi-threaded variants might also require GHUMVEE-ready versions of libstdc++ and libgomp. Once again, GHUMVEE ships with prebuilt versions of these libraries so if you're not planning to do anything crazy, **you can skip the rest of this section **.

If, for whatever reason, you want to build your own GHUMVEE-versions of these libraries, then use the following commands:

# download the gcc sources
apt-get source gcc-<latestversion>
cd gcc-<latestversion>
tar xJf gcc-*
cd gcc-<version>

# generate an up to date mvee_atomic.h header
/patch/to/ReMon/scripts/generate_atomic_header_new.sh > mvee_atomic.h

# patch libstdc++ and libgomp
patch -p1 < /path/to/ReMon/patches/libstdc++.<yourver>.patch
patch -p1 < /path/to/ReMon/patches/libgomp.<yourver>.patch

# make
./configure --enable-languages=c,c++ --disable-multilib
make -j 8

# "install" the libs
cp <arch>-pc-linux-gnu/libgomp/.libs/libgomp.so.1.0.0 /path/to/ReMon/patched_binaries/<arch>/libgomp/
cp <arch>-pc-linux-gnu/libstdc++-v3/src/.libs/libstdc++.so.6.0.<ver> /path/to/ReMon/patched_binaries/<arch>/libstdc++/

Known Issues

GCC emits code with ad-hoc synchronization to initialize local static variables. If these local statics get initialized in a multi-threaded context, this can cause mismatches (e.g. in PARSEC's raytrace benchmarks). This ad-hoc synchronization code can be eliminated using the -fno-threadsafe-statics compiler flag.

License

The IP-MON component is available under the licensing terms in IPMONLICENSE.txt. This license applies to the following files:

patches/linux-3.13-ipmon.patch
patches/linux-3.13-full-ipmon.patch
patches/linux-4.4.0-full-ipmon.patch
The entire IP-MON and IP-MON-atc folders

Some of the files in the exploits/BROP folder were downloaded from this website. The licensing terms for these files are unknown.

Unless otherwise specified, all other files in the repository are available under the licensing terms in GHUMVEELICENSE.txt.

MrBeanc/ReMon