Research impact of tb_flush after init_forkserver

[domenukk]

The current fork-server implementation on X64 might be negatively impacted by tb-flush in cpu-exec of unicorn:
https://github.com/unicorn-engine/unicorn/blob/0551b56633f658ec760eac54c14712d712b746d7/qemu/cpu-exec.c#

Problem:
To start the fork-server, a single insn is executed.
Afterwards, we exit the cpu loop and translated blocks are flushed.
Since the parent will wait for translation requests at this first insn, all pre-jitted blocks on future children may simply be flushed and re-jitted after input from AFL is read.

[domenukk]

Very unsuccessfully tried

index 94184a2..5f6e9d5 100755
--- a/qemu/cpu-exec.c
+++ b/qemu/cpu-exec.c
@@ -290,7 +299,13 @@ int cpu_exec(struct uc_struct *uc, CPUArchState *env)   // qq
     // Unicorn: flush JIT cache to because emulation might stop in
     // the middle of translation, thus generate incomplete code.
     // TODO: optimize this for better performance
+#if defined(UNICORN_AFL)
+    if (!env->uc->afl_area_ptr) {
+        tb_flush(env);
+    }
+#else
     tb_flush(env);
+#endif

-> The (normally working) testcase crashes with Invalid instruction (UC_ERR_INSN_INVALID) at address ffffffff83e00108 after the forkserver started.

[domenukk]

Found the root cause:

• After uc_emu_start was called for the second time in the child, the parent is still stuck in the first call to uc_emu_start, waiting to translate new basic blocks.
• The until address in the parent differs from the until address of the child.
• After the second fork, the child uses the cached translation block of the parent.
• As opposed to the first run in which the child translates the block itself, the second run will not stop at the correct until address (it would, instead, stop at the parent's until address).
• The child runs until it crashes somewhere (or loops forever)

As fix, I've started to add a uc_afl_forkserver_start function that can take a list of exits, see https://github.com/domenukk/unicorn/blob/55a7cec601f380226ec1e170f325310e7720e48c/bindings/python/unicornafl/unicorn.py#L376

This will be upstreamed to AFL++, the trickle into this repo.

[domenukk]

Merged to master with 3ac3007