Fast Check

Yet another property based testing framework BUT written in TypeScript

Getting started

Usage

Documentation [Generated TypeDoc]

Arbitraries
Custom arbitraries
Properties
Runners

Tips

Preview generated values
Replay after failure

Issues found by fast-check in famous packages

Examples - Examples against REST API - More examples

Getting started

Install the module with: npm install fast-check --save-dev

Running examples in RunKit: https://runkit.com/dubzzz/fast-check-basic-examples

Property based testing explained:

If you want to synchronize and build the code locally:

git clone https://github.com/dubzzz/fast-check.git && cd fast-check
npm install
npm run prebuild #generate missing implementations: tuple and properties
npm run build    #compile the code in ./src, build the ./lib content
npm run test     #run unit tests
npm run e2e      #run end-to-end tests: check the code can shrink on errors, replay on failure...

Usage

In mocha or jasmine

Using fast-check with mocha is really straightfoward. It can be used directly in describe, it blocks with no extra care.

The following snippets written in Javascript shows an example featuring two properties:

const fc = require('fast-check');

// Code under tests
const contains = (text, pattern) => text.indexOf(pattern) >= 0;

// Properties
describe('properties', () => {
    it('should always contain itself', () => {
        fc.assert(fc.property(fc.string(), text => contains(text, text)));
    });
    it('should always contain its substrings', () => {
        fc.assert(fc.property(fc.string(), fc.string(), fc.string(), (a,b,c) => contains(a+b+c, b)));
    });
});

In case of failure, the tests would raise a red flag and the output should help you to diagnose what went wrong in your implementation (example with a failing implementation of contain):

1) should always contain its substrings
    Property failed after 1 tests (seed: 1515709471288): [,,]
    Got error: Property failed by returning false

More on integration in: ava, jasmine, jest, mocha and tape.

In a web-page

In order to use fast-check from a web-page (for instance with QUnit or other testing tools), you have to reference the web-aware script as follow:

<script src="https://cdn.jsdelivr.net/npm/fast-check/lib/bundle.js"></script>

You can also reference a precise version by setting the version you want in the url:

<script src="https://cdn.jsdelivr.net/npm/fast-check@0.0.11/lib/bundle.js"></script>

Once it has been included, fast-check becomes accessible directly by calling fastcheck (in window.fastcheck). I highly recommend you to alias it by fc whenever possible by running const fc = fastcheck at the beginning of the scripts using it.

Documentation

Arbitraries

Arbitraries are responsible for the random generation (but deterministic) and shrink of datatypes. They can be combined together to build more complex datatypes.

Boolean (:boolean)

fc.boolean() either true or false

Numeric (:number)

Integer values:

fc.integer() all possible integers ie. from -2147483648 (included) to 2147483647 (included)
fc.integer(max: number) all possible integers between -2147483648 (included) and max (included)
fc.integer(min: number, max: number) all possible integers between min (included) and max (included)
fc.nat() all possible positive integers ie. from 0 (included) to 2147483647 (included)
fc.nat(max: number) all possible positive integers between 0 (included) and max (included)

Floating point numbers:

fc.float() uniformly distributed float value between 0.0 (included) and 1.0 (excluded)
fc.float(max: number) uniformly distributed float value between 0.0 (included) and max (excluded)
fc.float(min: number, max: number) uniformly distributed float value between min (included) and max (excluded)
fc.double()uniformly distributed double value between 0.0 (included) and 1.0 (excluded)
fc.double(max: number)uniformly distributed double value between 0.0 (included) and max (excluded)
fc.double(min: number, max: number)uniformly distributed double value between min (included) and max (excluded)

String (:string)

Single character only:

fc.hexa() one character in 0123456789abcdef (lower case)
fc.base64() one character in A-Z, a-z, 0-9, + or /
fc.char() between 0x20 (included) and 0x7e (included) , corresponding to printable characters (see https://www.ascii-code.com/)
fc.ascii() between 0x00 (included) and 0x7f (included)
fc.unicode() between 0x0000 (included) and 0xffff (included) but excluding surrogate pairs (between 0xd800 and 0xdfff). Generate any character of UCS-2 which is a subset of UTF-16 (restricted to BMP plan)
fc.char16bits() between 0x0000 (included) and 0xffff (included). Generate any 16 bits character. Be aware the values within 0xd800 and 0xdfff which constitutes the surrogate pair characters are also generated meaning that some generated characters might appear invalid regarding UCS-2 and UTF-16 encoding
fc.fullUnicode() between 0x0000 (included) and 0x10ffff (included) but excluding surrogate pairs (between 0xd800 and 0xdfff). Its length can be greater than one as it potentially contains multiple UTF-16 characters for a single glyph

Multiple characters:

fc.hexaString(), fc.hexaString(maxLength: number) or fc.hexaString(minLength: number, maxLength: number) string based on characters generated by fc.hexa()
fc.base64String(), fc.base64String(maxLength: number) or fc.base64String(minLength: number, maxLength: number) string based on characters generated by fc.base64(). Provide valid base64 strings: length always multiple of 4 padded with '=' characters. When using minLength and maxLength make sure that they are compatible together. For instance: asking for minLength=2 and maxLength=4 is impossible for base64 strings as produced by the framework
fc.string(), fc.string(maxLength: number) or fc.string(minLength: number, maxLength: number) string based on characters generated by fc.char()
fc.asciiString(), fc.asciiString(maxLength: number) or fc.asciiString(minLength: number, maxLength: number) string based on characters generated by fc.ascii()
fc.unicodeString(), fc.unicodeString(maxLength: number) or fc.unicodeString(minLength: number, maxLength: number) string based on characters generated by fc.unicode()
fc.string16bits(), fc.string16bits(maxLength: number) or fc.string16bits(minLength: number, maxLength: number) string based on characters generated by fc.char16bits(). Be aware that the generated string might appear invalid regarding the unicode standard as it might contain incomplete pairs of surrogate
fc.fullUnicodeString(), fc.fullUnicodeString(maxLength: number) or fc.fullUnicodeString(minLength: number, maxLength: number) string based on characters generated by fc.fullUnicode(). Be aware that the length is considered in terms of the number of glyphs in the string and not the number of UTF-16 characters
fc.stringOf(charArb: Arbitrary<string>), fc.stringOf(charArb: Arbitrary<string>, maxLength: number) or fc.stringOf(charArb: Arbitrary<string>, minLength: number, maxLength: number) string based on characters generated by charArb. minLength and maxLength define bounds of the number of elements generated using charArb that can be found in the final string

Strings that mimic real strings, with words and sentences:

json() or json(maxDepth: number) json strings having keys generated using fc.string(). String values are also produced by fc.string()
unicodeJson() or unicodeJson(maxDepth: number) json strings having keys generated using fc.unicodeString(). String values are also produced by fc.unicodeString()
fc.lorem(), fc.lorem(maxWordsCount: number) or fc.lorem(maxWordsCount: number, sentencesMode: boolean) lorem ipsum strings. Generator can be configured by giving it a maximum number of characters by using maxWordsCount or switching the mode to sentences by setting sentencesMode to true in which case maxWordsCount is used to cap the number of sentences allowed. This arbitrary is not shrinkable

Combinors of arbitraries (:T)

fc.constant<T>(value: T): Arbitrary<T> constant arbitrary only able to produce value: T
fc.constantFrom<T>(...values: T[]): Arbitrary<T> randomly chooses among the values provided. It considers the first value as the default value so that in case of failure it will shrink to it
fc.oneof<T>(...arbs: Arbitrary<T>[]): Arbitrary<T> randomly chooses an arbitrary at each new generation. Should be provided with at least one arbitrary. All arbitraries are equally probable and shrink is still working for the selected arbitrary. fc.oneof is able to shrink inside the failing arbitrary but not accross arbitraries (contrary to fc.constantFrom when dealing with constant arbitraries)
fc.option<T>(arb: Arbitrary<T>): Arbitrary<T | null> or fc.option<T>(arb: Arbitrary<T>, freq: number): Arbitrary<T | null> arbitrary able to nullify its generated value. When provided a custom freq value it changes the frequency of null values so that they occur one time over freq tries (eg.: freq=5 means that 20% of generated values will be null and 80% would be produced through arb). By default: freq=5
fc.array<T>(arb: Arbitrary<T>): Arbitrary<T[]>, fc.array<T>(arb: Arbitrary<T>, maxLength: number): Arbitrary<T[]> or fc.array<T>(arb: Arbitrary<T>, minLength: number, maxLength: number): Arbitrary<T[]> array of random length containing values generated by arb. By setting the parameters minLength and/or maxLength, the user can change the minimal (resp. maximal) size allowed for the generated array. By default: minLength=0 and maxLength=10
fc.set<T>(arb: Arbitrary<T>): Arbitrary<T[]>, fc.set<T>(arb: Arbitrary<T>, maxLength: number): Arbitrary<T[]> or fc.set<T>(arb: Arbitrary<T>, minLength: number, maxLength: number): Arbitrary<T[]> set of random length containing unique values generated by arb. All the values in the set are unique given the default comparator = (a: T, b: T) => a === b which can be overriden by giving another comparator function as the last argument on previous signatures
fc.tuple<T1,T2,...>(arb1: Arbitrary<T1>, arb2: Arbitrary<T2>, ...): Arbitrary<[T1,T2,...]> tuple generated by aggregating the values of arbX like generate: () => [arb1.generate(), arb2.generate(), ...]. This arbitrary perfectly handle shrinks and is able to shink on all the generators
fc.dictionary<T>(keyArb: Arbitrary<string>, valueArb: Arbitrary<T>): Arbitrary<{[Key:string]:T}> dictionary containing keys generated using keyArb and values gneerated by valueArb
fc.record<T>(recordModel: {[Key:string]: Arbitrary<T>}): Arbitrary<{[Key:string]: T}> or fc.record<T>(recordModel: {[Key:string]: Arbitrary<T>}, constraints: RecordConstraints): Arbitrary<{[Key:string]: T}> record using the incoming arbitraries to generate its values. It comes very useful when dealing with settings. It takes an optional parameter of type RecordConstraints to configure some of its properties. The setting withDeletedKeys=true instructs the record generator that it can omit some keys

Objects (:any)

The framework is able to generate totally random objects in order to adapt to programs that do not requires any specific data structure. All those custom types can be parametrized using ObjectConstraints.Settings.

export module ObjectConstraints {
    export interface Settings {
        maxDepth?: number;         // maximal depth allowed for this object
        key?: Arbitrary<string>;   // arbitrary for key
        values?: Arbitrary<any>[]; // arbitrary responsible for base value
    };
};

Default for key is: fc.string().

Default for values are: fc.boolean(), fc.integer(), fc.double(), fc.string() and constants among null, undefined, Number.NaN, +0, -0, Number.EPSILON, Number.MIN_VALUE, Number.MAX_VALUE , Number.MIN_SAFE_INTEGER, Number.MAX_SAFE_INTEGER, Number.POSITIVE_INFINITY or Number.NEGATIVE_INFINITY.

fc.anything() or fc.anything(settings: ObjectConstraints.Settings) generate a possible values coming from Settings and all objects or arrays derived from those same settings
fc.object() or fc.object(settings: ObjectConstraints.Settings) generate an object
fc.jsonObject() or fc.jsonObject(maxDepth: number) generate an object that is eligible to be stringified and parsed back to itself (object compatible with json stringify)
fc.unicodeJsonObject() or fc.unicodeJsonObject(maxDepth: number) generate an object with potentially unicode characters that is eligible to be stringified and parsed back to itself (object compatible with json stringify)

Custom arbitraries

Derive existing arbitraries

All generated arbitraries inherit from the same base class: Arbitrary.

It cames with two useful methods: filter(predicate: (t: T) => boolean): Arbitrary<T> and map<U>(mapper: (t: T) => U): Arbitrary<U>. These methods are used internally by the framework to derive some Arbitraries from existing ones.

Additionaly it comes with noShrink() which derives an existing Arbitrary<T> into the same Arbitrary<T> without the shrink option.

Filter values

filter(predicate: (t: T) => boolean): Arbitrary<T> can be used to filter undesirable values from the generated ones. It can be used as some kind of pre-requisite for the parameters required for your algorithm. For instance, you might need to generate two ordered integer values. One approach can be to use filter as follow:

const minMax = fc.tuple(fc.integer(), fc.integer())
                    .filter(t => t[0] < t[1]);

But be aware that using filter may highly impact the time required to generate a valid entry. In the previous example, half of the generated tuples will be rejected. It can nontheless be a very useful and powerful tool to derive your arbitraries quickly and easily.

Transform values

map<U>(mapper: (t: T) => U): Arbitrary<U> in its side does not filter any of the generated entries. It take one entry (generated or shrinked) and map it to another.

For instance the previous example could have been refactored as follow:

const minMax = fc.tuple(fc.integer(), fc.integer())
                    .map(t => t[0] < t[1] ? [t[0], t[1]] : [t[1], t[0]]);

Another example would be to derive fc.integer() and fc.array() to build fc.char() and fc.string():

const char = () => fc.integer(0x20, 0x7e).map(String.fromCharCode);
const string = () => fc.array(fc.char()).map(arr => arr.join(''));

Most of the built-in arbitraries use this trick to define themselves.

Remove the shrinker

Calling noShrink() on an Arbitrary<T> just remove the shrinker of the Arbitrary<T>. For instance, the following code will produce an Arbitrary<number> without shrinking operation.

const intNoShrink = fc.integer().noShrink();

Build your own

You can also fully customize your arbitrary and by not deriving it from any of the buit-in arbitraries. What you have to do is to derive from Arbitrary and implement generate(mrng: Random): Shrinkable<T>.

generate is responsable for the generation of one new random entity of type T (see signature above). In order to fulfill it in a deterministic way it received a mrng: Random. It comes with multiple built-in helpers to generate values:

next(n: number): number: uniformly distributed n bits value (max value of n = 31)
nextBoolean(): boolean: uniformly distributed boolean value
nextInt(): number: uniformly distributed integer value
nextInt(from: number, to: number): number: uniformly distributed integer value between from (inclusive) and to (inclusive)
nextDouble(): number: uniformly distributed double value between 0.0 (included) and 1.0 (not included)

The generated value also came with a shrink method able to derive smaller values in case of failure. It can be ignored making the arbitrary not shrinkable.

Once again the built-in types can be very helpful if you need an example.

Properties

fc.property: define a new property ie. a list of arbitraries and a test function to assess the success

The predicate would be considered falsy if its throws or if output evaluates to false.

function property<T1>(
        arb1: Arbitrary<T1>,
        predicate: (t1:T1) => (boolean|void)): Property<[T1]>;
function property<T1,T2>(
        arb1: Arbitrary<T1>, arb2: Arbitrary<T2>,
        predicate: (t1:T1,t2:T2) => (boolean|void)): Property<[T1,T2]>;
...

fc.asyncProperty: define a new property ie. a list of arbitraries and an asynchronous test function to assess the success

The predicate would be considered falsy if its throws or if output evaluates to false (after await).

function asyncProperty<T1>(
        arb1: Arbitrary<T1>,
        predicate: (t1:T1) => Promise<boolean|void>): AsyncProperty<[T1]>;
function asyncProperty<T1,T2>(
        arb1: Arbitrary<T1>, arb2: Arbitrary<T2>,
        predicate: (t1:T1,t2:T2) => Promise<boolean|void>): AsyncProperty<[T1,T2]>;
...

WARNING:

The predicate function must not change the inputs it received. If it needs to, it has to clone them before going on. Impacting the inputs might led to bad shrinking and wrong display on error.

Nonetheless a failing property will still be a failing property.

Runners

fc.assert: run the property and throws in case of failure

This function has to be awaited in case it is called on an asynchronous property.

This function is ideal to be called in describe, it blocks. It does not return anything in case of success.

It can be parametrized using its second argument.

export interface Parameters {
    seed?: number;      // optional, initial seed of the generator: Date.now() by default
    numRuns?: number;   // optional, number of runs before success: 100 by default 
    timeout?: number;   // optional, only taken into account for asynchronous runs (asyncProperty)
                        // specify a timeout in milliseconds, maximum time for the predicate to return its result
                        // only works for async code, will not interrupt a synchronous code: disabled by default
    path?: string;      // optional, way to replay a failing property directly with the counterexample
                        // it can be fed with the counterexamplePath returned by the failing test (requires seed too)
    logger?: (v: string) => void; // optional, log output: console.log by default
    unbiased?: boolean; // optional, force the use of unbiased arbitraries: biased by default
    verbose?: boolean;  // optional, enable verbose mode: false by default
                        // when enabling verbose mode,
                        // you will be provided the list of all failing entries encountered
                        // whenever a property fails - useful to detect patterns
}

function assert<Ts>(property: IProperty<Ts>, params?: Parameters);

fc.check: run the property and return an object containing the test status along with other useful details

This function has to be awaited in case it is called on an asynchronous property.

It should never throw whatever the status of the test.

It can be parametrized with the same parameters than fc.assert.

The details returned by fc.check are the following:

interface RunDetails<Ts> {
    failed: boolean,         // false in case of failure, true otherwise
    numRuns: number,         // number of runs (all runs if success, up and including the first failure if failed)
    numShrinks: number,      // number of shrinks (depth required to get the minimal failing example)
    seed: number,            // seed used for the test
    counterexample: Ts|null, // failure only: shrunk conterexample causig the property to fail
    counterexamplePath: string|null, // failure only: the exact path to re-run the counterexample
                                     // In order to replay the failing case directly,
                                     // this value as to be set as path attribute in the Parameters (with the seed)
                                     // of assert, check, sample or even statistics
    error: string|null,      // failure only: stack trace and error details
}

function check<Ts>(property: IProperty<Ts>, params?: Parameters);

fc.sample: sample generated values of an Arbitrary<T> or Property<T>

It builds an array containing all the values that would have been generated for the equivalent test.

It also accept Parameters as configuration in order to help you diagnose the shape of the inputs that will be received by your property.

type Generator<Ts> = Arbitrary<Ts> | IProperty<Ts>;

function sample<Ts>(generator: Generator<Ts>): Ts[];
function sample<Ts>(generator: Generator<Ts>, params: Parameters): Ts[];
function sample<Ts>(generator: Generator<Ts>, numGenerated: number): Ts[];

fc.statistics: classify the values produced by an Arbitrary<T> or Property<T>

It provides useful statistics concerning generated values. In order to be able to gather those statistics it has to be provided with a classifier function that can classify the generated value in zero, one or more categories (free labels).

It also accept Parameters as configuration in order to help you diagnose the shape of the inputs that will be received by your property.

Statistics are dumped into console.log but can be redirected to another source by modifying the logger key in Parameters.

type Generator<Ts> = Arbitrary<Ts> | IProperty<Ts>;
type Classifier<Ts> = ((v: Ts) => string) | ((v: Ts) => string[]);

function statistics<Ts>(generator: Generator<Ts>, classify: Classifier<Ts>): void;
function statistics<Ts>(generator: Generator<Ts>, classify: Classifier<Ts>, params: Parameters): void;
function statistics<Ts>(generator: Generator<Ts>, classify: Classifier<Ts>, numGenerated: number): void;

Tips

Preview generated values

Before writing down your test, it might be great to confirm that the arbitrary you will be using produce the values you want.

This can be done very easily by using either fc.sample or fc.statistics.

The following code constructs an array containing the first 10 values that would have been generated by the arbitrary fc.anything() if used inside a fc.assert or fc.check:

fc.sample(
    fc.anything(), // arbitrary or property to extract the values from
    10             // number of values to extract
);

In some cases, having a sample is not enough and we want more insights about the generated data. For instance, I might be interested by the share of even numbers generated by fc.nat(). For that purpose I can use fc.statistics as follow:

fc.statistics(
    fc.nat(),    // arbitrary or property to extract the values from
    n => n % 2 === 0 ? 'Even number' : 'Odd number', // classifier
    10000        // number of values to extract
);
// Possible output (console.log):
// Odd number...50.30%
// Even number..49.70%

Replay after failure

fast-check comes with a must have feature: replay a failing case immediately given its seed and path (seed only to replay all).

Whenever fc.assert encounters a failure, it displays an error log featuring both the seed and the path to replay it. For instance, in the output below the seed is 1525890375951 and the path 0:0.

Error: Property failed after 1 tests (seed: 1525890375951, path: 0:0): [0]
Shrunk 1 time(s)
Got error: Property failed by returning false

In order to replay the failure on the counterexample - [0], you have to change your code as follow:

// Original code
fc.assert(
  fc.property(
    fc.nat(),
    checkEverythingIsOk
  )
);

// Replay code: straight to the minimal counterexample
fc.assert(
  fc.property(
    fc.nat(),
    checkEverythingIsOk
  ),
  {
    seed: 1525890375951,
    path: "0:0"
  }
);

Issues found by fast-check in famous packages

fast-check has been able to find some unexpected behaviour among famous npm packages. Here are some of the errors detected using fast-check:

js-yaml

Issue detected: enabling !!int: binary style when dumping negative integers produces invalid content [more]

Code example: yaml.dump({toto: -10}, {styles:{'!!int':'binary'}}) produces toto: 0b-1010 not toto: -0b1010

left-pad

Issue detected: unicode characters outside of the BMP plan are not handled consistently [more]

Code example:

leftPad('a\u{1f431}b', 4, 'x') //=> 'a\u{1f431}b'  -- in: 3 code points, out: 3 code points
leftPad('abc', 4, '\u{1f431}') //=> '\u{1f431}abc' -- in: 3 code points, out: 4 code points

bitjson/fast-check

Fast Check

Yet another property based testing framework BUT written in TypeScript

Getting started

Usage

In mocha or jasmine

In a web-page

Documentation

Arbitraries

Boolean (:boolean)

Numeric (:number)

String (:string)

Combinors of arbitraries (:T)

Objects (:any)

Custom arbitraries

Derive existing arbitraries

Filter values

Transform values

Remove the shrinker

Build your own

Properties

Runners

Tips

Preview generated values

Replay after failure

Issues found by fast-check in famous packages

js-yaml

left-pad