Fast and lazy collection operations.
Working with methods like .map()
, .filter()
and .reduce()
is nice,
however they create new arrays and everything is eagerly done before going to
the next step.
This is where lazy collections come in, under the hood we use iterators and async iterators so that your data flows like a stream to have the optimal speed.
All functions should work with both iterator
and asyncIterator
, if one of
the functions uses an asyncIterator
(for example when you introduce
delay(100)
), don't forget to await
the result!
const program = pipe(
map(x => x * 2),
filter(x => x % 4 === 0),
filter(x => x % 100 === 0),
filter(x => x % 400 === 0),
toArray()
);
program(range(0, 1000000));
⚠️ This is not a scientific benchmark, there are flaws with this. This is just meant to showcase the power of lazy-collections.
Lazy | Eager | ||
---|---|---|---|
Duration | 2.19ms |
1.29s |
589x faster |
Memory heapTotal | 9.48 MB |
297.96 MB |
31x less memory |
Memory heapUsed | 5.89 MB |
265.46 MB |
45x less memory |
Memory data collected using: http://nodejs.org/api/process.html#process_process_memoryusage
import {
pipe,
range,
filter,
takeWhile,
slice,
toArray,
} from 'lazy-collections';
// Lazy example
const program = pipe(
range(0, 10_000_000),
filter(x => x % 100 === 0),
filter(x => x % 4 === 0),
filter(x => x % 400 === 0),
takeWhile(x => x < 1_000),
slice(0, 1_000),
toArray()
);
program(); // [ 0, 400, 800 ]
// Eager example
function program() {
return (
// Equivalent of the range()
[...new Array(10_000_000).keys()]
.filter(x => x % 100 === 0)
.filter(x => x % 4 === 0)
.filter(x => x % 400 === 0)
// Equivalent of the takeWhile
.reduce((acc, current) => {
return current < 1_000 ? (acc.push(current), acc) : acc;
}, [])
.slice(0, 1_000)
);
}
program(); // [ 0, 400, 800 ]
This is actually a stupid non-real-world example. However, it is way more
efficient at doing things. That said, yes you can optimize the eager example
way more if you want to. You can combine the filter
/ reduce
/ ...
. However,
what I want to achieve is that we can have separated logic in different filter
or map
steps without thinking about performance bottlenecks.
We can use compose to compose functions together and return a new function which combines all other functions.
import { compose } from 'lazy-collections';
// Create a program (or a combination of functions)
const program = compose(fn1, fn2, fn3);
program();
// fn1(fn2(fn3()))
We can use pipe to compose functions together and return a new function which combines all other functions.
The difference between pipe
and compose
is the order of execution of the
functions.
import { pipe } from 'lazy-collections';
// Create a program (or a combination of functions)
const program = pipe(fn1, fn2, fn3);
program();
// fn3(fn2(fn1()))
Concat multiple iterators or arrays into a single iterator.
import { pipe, concat, toArray } from 'lazy-collections';
const program = pipe(
concat([0, 1, 2], [3, 4, 5], [6, 7, 8], [9, 10]),
toArray()
);
program();
// [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ]
Should return true if all values match the predicate.
import { pipe, every } from 'lazy-collections';
const program = pipe(every(x => x === 2));
program([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
// false
Filter out values that do not meet the condition.
import { pipe, filter, toArray } from 'lazy-collections';
const program = pipe(
filter(x => x % 2 === 0),
toArray()
);
program([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
// [ 2, 4, 6, 8, 10 ]
Find a value based on the given predicate.
import { pipe, find } from 'lazy-collections';
const program = pipe(find(x => x === 2));
program([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
// 2
Find an index based on the given predicate.
import { pipe, findIndex } from 'lazy-collections';
const program = pipe(findIndex(x => x === 2));
program([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
// 2
Map a value from A to B.
import { pipe, map, toArray } from 'lazy-collections';
const program = pipe(
map(x => x * 2),
toArray()
);
program([1, 2, 3]);
// [ 2, 4, 6 ]
Reduce the data to a single value.
import { pipe, reduce } from 'lazy-collections';
const program = pipe(reduce((total, current) => total + current, 0));
program([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
// 55
Reverses the iterator.
note: This is currently very slow because it has to go through the full iterator first!
import { pipe, reverse, toArray } from 'lazy-collections';
const program = pipe(range(0, 5), reverse(), toArray());
program();
// [ 5, 4, 3, 2, 1, 0 ]
Should return true if some of the values match the predicate.
import { pipe, some } from 'lazy-collections';
const program = pipe(some(x => x === 2));
program([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
// true
Alias:
mean
Gets the average of number of values.
import { pipe, average, toArray } from 'lazy-collections';
const program = pipe(average());
program([6, 7, 8, 9, 10]);
// 8
Find the maximum value of the given list
import { pipe, range, max } from 'lazy-collections';
const program = pipe(range(0, 5), max());
program();
// 5
Find the minimum value of the given list
import { pipe, range, min } from 'lazy-collections';
const program = pipe(range(5, 10), min());
program();
// 5
Should sum an array or iterator.
import { pipe, sum } from 'lazy-collections';
const program = pipe(sum());
program([1, 1, 2, 3, 2, 4, 5]);
// 18
Chunk the data into pieces of a certain size.
import { pipe, chunk, toArray } from 'lazy-collections';
const program = pipe(chunk(3), toArray());
program([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
// [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ], [ 10 ] ];
Filters out all falsey values.
import { pipe, compact, toArray } from 'lazy-collections';
const program = pipe(compact(), toArray());
program([0, 1, true, false, null, undefined, '', 'test', NaN]);
// [ 1, true, 'test' ];
Will make he whole program async. It will add a delay of x milliseconds when an item goes through the stream.
import { pipe, range, delay, map, toArray } from 'lazy-collections';
const program = pipe(
range(0, 4),
delay(5000), // 5 seconds
map(() => new Date().toLocaleTimeString()),
toArray()
);
await program();
// [ '10:00:00', '10:00:05', '10:00:10', '10:00:15', '10:00:20' ];
By default we will flatten recursively deep.
import { pipe, flatten, toArray } from 'lazy-collections';
const program = pipe(flatten(), toArray());
program([1, 2, 3, [4, 5, 6, [7, 8], 9, 10]]);
// [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ]
But you can also just flatten shallowly
import { pipe, flatten, toArray } from 'lazy-collections';
const program = pipe(flatten({ shallow: true }), toArray());
program([1, 2, 3, [4, 5, 6, [7, 8], 9, 10]]);
// [ 1, 2, 3, 4, 5, 6, [ 7, 8 ], 9, 10 ]
Generate accepts a function that function will be called over and over again.
Don't forget to combine this with a function that ensures that the data stream
will end. For example, you can use take
, takeWhile
or slice
.
import { pipe, generate, take, toArray } from 'lazy-collections';
const program = pipe(generate(Math.random), take(3), toArray());
program();
// [ 0.7495421596380878, 0.09819118640607383, 0.2453718461872143 ]
Groups the iterator to an object, using the keySelector function.
import { pipe, groupBy, range } from 'lazy-collections';
// A function that will map the value to the nearest multitude. In this example
// we will map values to the nearest multitude of 5. So that we can group by
// this value.
function snap(multitude: number, value: number) {
return Math.ceil(value / multitude) * multitude;
}
const program = pipe(
range(0, 10),
groupBy((x: number) => snap(5, x))
);
program();
// {
// 0: [0],
// 5: [1, 2, 3, 4, 5],
// 10: [6, 7, 8, 9, 10],
// }
Alias:
first
Gets the first value of the array / iterator. Returns undefined
if there is no
value.
import { pipe, chunk, toArray } from 'lazy-collections';
const program = pipe(head());
program([6, 7, 8, 9, 10]);
// 6
Partition data into 2 groups based on the predicate.
import { pipe, partition, range, toArray } from 'lazy-collections';
const program = pipe(
range(1, 4),
partition(x => x % 2 !== 0),
toArray()
);
program();
// [ [ 1, 3 ], [ 2, 4 ] ]
Create a range of data using a lowerbound, upperbound and step. The step is
optional and defaults to 1
.
import { pipe, range, toArray } from 'lazy-collections';
const program = pipe(range(5, 20, 5), toArray());
program();
// [ 5, 10, 15, 20 ]
Allows you to skip X values of the input.
import { pipe, range, skip, toArray } from 'lazy-collections';
const program = pipe(range(0, 10), skip(3), toArray());
program();
// [ 4, 5, 6, 7, 8, 9, 10 ]
Slice a certain portion from your data set. It accepts a start index and an end index.
import { pipe, range, slice, toArray } from 'lazy-collections';
const program = pipe(range(0, 10), slice(3, 5), toArray());
program();
// [ 3, 4, 5 ]
// Without the slice this would have generated
// [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ]
Allows you to take X values of the input.
import { pipe, range, take, toArray } from 'lazy-collections';
const program = pipe(range(0, 10), take(3), toArray());
program();
// [ 1, 2, 3 ]
This is similar to take
, but instead of a number as a value it takes a
function as a condition.
import { pipe, range, takeWhile, toArray } from 'lazy-collections';
const program = pipe(
range(0, 10),
takeWhile(x => x < 5),
toArray()
);
program();
// [ 0, 1, 2, 3, 4 ]
Allows you to tap into the stream, this way you can intercept each value.
import { pipe, range, tap, toArray } from 'lazy-collections';
const program = pipe(
range(0, 5),
tap(x => {
console.log('x:', x);
}),
toArray()
);
program();
// x: 0
// x: 1
// x: 2
// x: 3
// x: 4
// x: 5
// [ 0, 1, 2, 3, 4, 5 ]
Converts an array or an iterator to an actual array.
import { pipe, range, toArray } from 'lazy-collections';
const program = pipe(range(0, 10), toArray());
program();
// [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ]
Make your data unique.
import { pipe, unique, toArray } from 'lazy-collections';
const program = pipe(unique(), toArray());
program([1, 1, 2, 3, 2, 4, 5]);
// [ 1, 2, 3, 4, 5 ]
Filter out values based on the given properties.
import { pipe, where, range, map, where, toArray } from 'lazy-collections';
const program = pipe(
range(15, 20),
map(age => ({ age })),
where({ age: 18 }),
toArray()
);
program();
// [ { age: 18 } ]
Zips multiple arrays / iterators together.
import { pipe, zip, toArray } from 'lazy-collections';
const program = pipe(zip(), toArray());
program([
[0, 1, 2],
['A', 'B', 'C'],
]);
// [ [ 0, 'A' ], [ 1, 'B' ], [ 2, 'C' ] ]