Streams can be hard to test.
You want to test transform streams etc with large batches of data to ensure they're robust, but you don't want to include big files to test with in the repo.
This module provides:
- Readable stream which streams out data of whatever size you need
- Writable stream which receives data and checks it's what it should be
This is achieved entirely in memory (no file streams etc) but with minimal memory use.
The streams are created from a generator function you provide, which can produce streams of whatever content and size you require.
npm install stream-gen
Readable stream that gets its content from a generator.
The generator must yield
a byte value (0-255) each time it is called. The content of the stream is made up of these bytes.
const { GeneratorReadStream } = require('stream-gen');
// Data generator - creates 1KB of data
function* gen() {
for (let i = 0; i < 1024; i++) {
yield i % 256;
}
}
const producer = new GeneratorReadStream( gen );
producer.pipe( destination );
You can also use a generator which produces 2-byte or 4-byte values (i.e. 16 bit or 32 bit) using options.byte
.
Writable stream that receives content and compares it to result of a generator. It calls callback
with result of the comparison.
The generator must be of same type as for GeneratorReadStream
.
const { GeneratorWriteStream } = require('stream-gen');
const checker = new GeneratorWriteStream( gen, function(err) {
if (err) return console.log('Stream differed');
console.log('Stream as expected');
} );
source.pipe( checker );
The two parts work together to test that a transfer of data has been completed without altering the data.
Let's say you are testing a lossless compression component:
Input -> Compress -> Decompress -> Output
Input and output should be the same.
You can test this with:
const { GeneratorReadStream, GeneratorWriteStream } = require('stream-gen');
const zlib = require('zlib');
// Data generator - creates 100MB of data
function* gen() {
for (let i = 0; i < 100 * 1024 * 1024; i++) {
yield i % 256;
}
}
// Create producer and checker
const producer = new GeneratorReadStream( gen ),
const checker = new GeneratorWriteStream( gen, function(err) {
if (err) throw err;
console.log('It works!');
} );
// Create compressor and decompressor
const compressor = zlib.createDeflate();
const decompressor = zlib.createInflate();
// Pipe them all together
producer.pipe(compressor).pipe(decompressor).pipe(checker);
The callback on checker
tells us if data after compression and decompression is the same at it started off as.
There we go! We tested ZLIB on 100MB of data, using hardly any memory and no disc access.
To use a generator which outputs 16 bit or 32 bit numbers rather than 8 bit, set options.bytes
to 2
or 4
.
Default is 1
(i.e. 8 bit).
function* gen() {
for (let i = 0; i < 100 * 1024 * 1024; i++) {
yield i % (256 * 256);
}
}
const stream = new GeneratorReadStream( gen, { bytes: 2 } );
maxSize
option determines largest size chunk stream will emit.
const stream = new GeneratorReadStream( gen, { maxSize: 1024 } );
Passed to Node's Stream constructor.
Passed to Node's Stream constructor.
Use npm test
to run the tests. Use npm run cover
to check coverage.
See changelog.md
If you discover a bug, please raise an issue on Github. https://github.com/overlookmotel/stream-gen/issues
Pull requests are very welcome. Please:
- ensure all tests pass before submitting PR
- add an entry to changelog
- add tests for new features
- document new functionality/API additions in README