concu
A simple method wrapper for concurrent processing of an array in javascript.
See introduction blog post on the Locai Engineering Blog
Introduction
The Concu library helps you processing series of data (typically an array) by chunking your data in smaller bits and concurrently processing these chunks. This is typically useful if you have the possibility of processing data in parallel, however you want to have control over:
- how many concurrent executions
- size of the batch being processed
Limiting concurrency and batch size may be needed because of downstream I/O dependencies.
When opted for concurrency of 1, the method presents a simple sequential execution on partial data chunks.
The original focus was on helping to process large CSV files for which each row had a lot of IO, and each row could be processed independently of any other row.
Internals
Concu is basically a simple wrapper around around pLimit and Promise.all()
Usage
Typescript example
import { concu } from "concu"
// create an array with 10000 elements
const array = Array(1000);
const dataToProcess: number[] = [...array.keys()];
// Dummy method to just calculate the sum
const processData = (numbers: number[]): number => {
return numbers.reduce( (sum, val) => sum+val, 0);
}
const concurrency = 10; // number of parallel executions
const chunkSize = 100; // size of every chunk
const run = async (): Promise<number[]> => {
return await concu(processData, concurrency, chunkSize, dataToProcess)
};
run().then( (result: number[]) => {
const sum = result.reduce( (total, value) => total + value, 0);
console.log(sum);
});
API
concu(fun, chunkSize, concurrency, data, ...args)
returns a Promise resolving to an array of individual results of each individual execution
fun: (data: any[], ...args: any[]) => any
function which accepts a data array and other possible arguments.
chunkSize: number
How many elements of the array to be processed
concurrency: number
how many executions in parallel
...args: any[]
optional arguments that will be passed to fun
Acknowledgments
This work has been created at Locai Solutions as part of the data import efforts.