[question] Does async reading work ? For await sub function.
Closed this issue · 6 comments
Thanks for this great job.
Reading stream is pretty good !
Does async reading work ?
If i must to do this :
import fs from 'fs';
import CsvReadableStream from 'csv-reader'
let inputStream = fs.createReadStream('my_data.csv', 'utf8');
inputStream
.pipe(new CsvReadableStream({ parseNumbers: true, parseBooleans: true, trim: true }))
.on('data', async (row) => {
console.log('A row arrived: ', row);
await makeMyDreamReality(row) // does await work ?
})
.on('end', function () {
console.log('No more rows!');
});
Ok, is there a mechanism to use the read stream in asynchronous (promise) mode ?
Without necessarily modifying the library.
Maybe a track : https://nodejs.org/api/stream.html#streams-promises-api
I'm trying...
I find trick (thanks Chat GPT), work on other csv library (like csv-parser), not this.
const fs = require('fs');
const CsvReader = require('csv-reader');
async function processLine(line) {
// async call
console.log(line);
}
async function processCsvFile(filePath) {
const csvReader = new CsvReader({
headers: true,
skipEmptyLines: true,
});
const stream = fs.createReadStream(filePath);
stream.pipe(csvReader);
csvReader.on('data', (line) => {
stream.pause();
processLine(line)
.then(() => {
stream.resume();
})
.catch((err) => {
console.error('error:', err);
});
});
csvReader.on('end', () => {
console.log('end CSV.');
});
}
processCsvFile('file.csv')
.then(() => {
console.log('end read.');
})
.catch((err) => {
console.error('error:', err);
});
pause/resume should work on any stream, but this is a transformer stream! You need to pause the source stream.
So if you did this:
const stream = fs.createReadStream(filePath);
stream.pipe(csvReader);
You need to pause stream and not csvReader.
I don't understand.
On my example, the pause is actually on the stream: stream.pause();
Have you try this exemple ?
Okay I've just tested, and this does not work, even with csv-parser.
.pipe() does not pipe the pause events.
When you pause the source stream, you cause it to not read any more from the file, but a chunk was read already and continues to be parsed by the transform stream (csv-reader).
If you want it to not emit any more data even though it has been read already, you need to manually call pause() on the csv-reader as well.
This can't be done from within the library as it does not have a reference to the source stream so there's no knowledge of whether it has been paused.