Fuzzlyn
Fuzzlyn is a fuzzer which utilizes Roslyn to generate random C# programs. It runs these programs on .NET core and ensures that they give the same results when compiled in debug and release mode.
We developed Fuzzlyn as a project for the 2018 Language-Based Security course at Aarhus University. Using Fuzzlyn, we have found and reported several bugs in RyuJIT used both by .NET Core and the full .NET framework. We have also found and reported bugs in Roslyn itself.
Bugs reported
We have reported the following bugs:
- NullReferenceException thrown for multi-dimensional arrays in release (fixed)
- Wrong integer promotion in release (fixed)
- Cast to ushort is dropped in release (fixed)
- Wrong value passed to generic interface method in release
- Constant-folding int.MinValue % -1
- Deterministic program outputs indeterministic results on Linux in release (fixed)
- RyuJIT incorrectly reorders expression containing a CSE, resulting in exception thrown in release
- RyuJIT incorrectly narrows value on ARM32/x86 in release (fixed)
- Invalid value numbering when morphing casts that changes signedness after global morph (fixed)
- RyuJIT spills 16 bit value but reloads as 32 bits in ARM32/x86 in release
- RyuJIT fails to preserve variable allocated to RCX around shift on x64 in release (fixed)
- RyuJIT: Invalid ordering when assigning ref-return (fixed)
- RyuJIT: Argument written to stack too early on Linux
- RyuJIT: Morph forgets about side effects when optimizing casted shift
- RyuJIT: By-ref assignment with null leads to runtime crash (fixed)
- RyuJIT: Mishandling of subrange assertion for rewritten call parameter
- RyuJIT: Incorrect ordering around Interlocked.Exchange and Interlocked.CompareExchange
- RyuJIT: Missing zeroing of upper bits for small struct used in Volatile.Read
- RyuJIT: Incorrect 4-byte immediate emitted for shift causes access violation
- Finally block belonging to unexecuted try runs anyway
Fuzzlyn has found many thousands of programs producing deviating behavior. Some of the first examples we found can be seen in the examples folder in the v1.0 tag (most of these have since been fixed). To take a couple of them, Fuzzlyn automatically found and produced the following programs:
// Generated by Fuzzlyn on 2018-06-03 16:17:09
// Seed: 10744458083861091494
// Reduced from 21.3 KB to 0.2 KB
// Debug: Outputs '246'
// Release: Outputs '4294967286'
public class Program
{
static sbyte s_1 = -10;
public static void Main()
{
ulong vr44 = (byte)(0U ^ s_1);
System.Console.WriteLine(vr44);
}
}
// Generated by Fuzzlyn on 2018-06-03 16:15:22
// Seed: 10187462581749713401
// Reduced from 186.5 KB to 0.2 KB
// Debug: Runs successfully
// Release: Throws 'System.DivideByZeroException'
public class Program
{
public static void Main()
{
var vr219 = 'N' % ((35815 / M1(new ushort[]{65535})) | 1);
}
static ushort M1(ushort[] arg2)
{
return arg2[0];
}
}
We are working on isolating which bugs are distinct from the ones already reported.
Supported constructs
Fuzzlyn generates only a limited subset of C#. Most importantly, it does not support loops yet. It supports structs and classes, though it does not generate member methods in these. We make no attempt to fully support all kinds of expressions and statements.
Using Fuzzlyn to find bugs
To run Fuzzlyn you must specify the number of programs to generate and check. To run a Fuzzlyn instance that generates a million programs and tries to find bugs in these, run:
dotnet fuzzlyn.dll --num-programs 1000000
Note that this only works with .NET core (i.e. a host like dotnet
must be running the program). This is because Fuzzlyn runs instances of itself and it uses the dotnet
that started Fuzzlyn to do this.
This command will not produce any output to stdout. However, when a program with deviating behavior is found, Fuzzlyn will append its seed and information about its execution to a file in the current directory called Execution_Mismatch.txt
.
Regenerating full programs
When a seed has been obtained, the full program can be regenerated by doing the following:
dotnet fuzzlyn.dll --seed <seed here> --output-source
This will generate exactly the code that produced deviating behavior at runtime. However, these examples cannot be run directly because they include checksumming of variables, which requires the IRuntime
interface to be passed to the Main
method. Instead, you can disable checksumming by passing the --checksum-
(note the minus) switch:
dotnet fuzzlyn.dll --seed <seed here> --output-source --checksum-
These examples are not very useful because they are very big. For that reason, Fuzzlyn includes an automatic reducer, which takes a seed and reduces the program specified by that seed to something smaller, while retaining the interesting behavior.
Reducing programs
To reduce a program, use the --reduce
switch:
dotnet fuzzlyn.dll --seed <seed here> --reduce
Depending on the size of the program, this will take a while (the biggest example in the examples
directory takes roughly 10 minutes to reduce on an i7-4770k). However, most programs do not take longer than a couple of minutes. The output of this command will be a small C# program, that includes information about its seed, size and runtime behavior.
Reproducing errors in reduced programs
The reduced programs produced by the --reduce
switch are not the original reduced programs that caused deviating behavior. Most specifically, the programs have had their checksum calls replaced by System.Console.WriteLine
calls, and other checksumming related code removed. This means that the files can be compiled directly with csc.exe
, or pasted into VS directly. However, it also means that in very rare cases, the examples do not actually reproduce the errors they are supposed to. One such case is the following:
// Generated by Fuzzlyn on 2018-06-03 16:15:22
// Seed: 1019504228635510285
// Reduced from 154.8 KB to 0.9 KB
// Debug: Outputs '10402227607262999317'
// Release: Throws 'System.NullReferenceException'
struct S0
{
public uint F0;
public char F1;
public int F2;
public int F3;
public uint F4;
public ulong F5;
public short F6;
public byte F7;
public S0(uint f0, char f1, int f2, int f3, uint f4, ulong f5, short f6, byte f7)
{
F0 = f0;
F1 = f1;
F2 = f2;
F3 = f3;
F4 = f4;
F5 = f5;
F6 = f6;
F7 = f7;
}
}
public class Program
{
static int[, ] s_1 = new int[, ]{{-1860696896}};
public static void Main()
{
var vr196 = new S0(0U, 'Y', 0, 876181050, 2606508611U, 10402227607262999317UL, 15399, 254);
var vr197 = (uint)(0U & s_1[0, 0]);
var vr198 = s_1[0, 0];
M20(vr196, vr197);
}
static int M20(S0 arg0, uint arg2)
{
System.Console.WriteLine(arg0.F5);
return -2147483647;
}
}
In these cases it is easy to introduce the deviating behavior by reintroducing some of the code and class hierarchies used by the checksumming:
// Generated by Fuzzlyn on 2018-06-03 16:15:22
// Seed: 1019504228635510285
// Reduced from 154.8 KB to 0.9 KB
// Debug: Outputs '10402227607262999317'
// Release: Throws 'System.NullReferenceException'
struct S0
{
public uint F0;
public char F1;
public int F2;
public int F3;
public uint F4;
public ulong F5;
public short F6;
public byte F7;
public S0(uint f0, char f1, int f2, int f3, uint f4, ulong f5, short f6, byte f7)
{
F0 = f0;
F1 = f1;
F2 = f2;
F3 = f3;
F4 = f4;
F5 = f5;
F6 = f6;
F7 = f7;
}
}
interface IRuntime
{
void WriteLine<T>(T val);
}
class Runtime : IRuntime
{
public void WriteLine<T>(T val) => System.Console.WriteLine(val);
}
public class Program
{
static IRuntime s_rt;
static int[, ] s_1 = new int[, ]{{-1860696896}};
public static void Main()
{
s_rt = new Runtime();
var vr196 = new S0(0U, 'Y', 0, 876181050, 2606508611U, 10402227607262999317UL, 15399, 254);
var vr197 = (uint)(0U & s_1[0, 0]);
var vr198 = s_1[0, 0];
M20(vr196, vr197);
}
static int M20(S0 arg0, uint arg2)
{
s_rt.WriteLine(arg0.F5);
return -2147483647;
}
}
While this is still not exactly the same as if Fuzzlyn generated it, it reproduces the error described at the start of the file.