batchSize bigger than 1 => "TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined"
Opened this issue · 1 comments
PlkMarudny commented
When I try to use queue in a batch mode, any batchSize > 1 causes tasks to fail with an error:
TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
The setup is minimal:
const downloadFile = async (fileUrl, downloadFolder, fileName) => {
const localFilePath = path.resolve(__dirname, downloadFolder, fileName);
try {
const response = await axios({
method: "GET",
url: fileUrl,
responseType: "stream",
});
await response.data.pipe(fs.createWriteStream(localFilePath));
return fileName;
} catch (err) {
throw (err);
}
};
const cb = (arg) => { console.log(arg); };
const processData = (data, cb) => {
downloadFile(data.url, data.destFolder, data.destFilename).then((result) =>
cb(null, `file ${result} downloaded`)).catch((err) => {
// console.log(err);
cb(err, null);
});
};
const jobQueue = new Queue(processData, { batchSize: 2, concurrent: 1 });
const data1 = {
id: 1,
url: "https://images.unsplash.com/photo-1616548479562-d5a581c064f4?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=2125&q=80",
destFolder: "D:\\images",
destFilename: "1.jpg"
};
// this one should fail
const data2 = {
id: 2,
url: "https://images.unsplash.co/photo-1550389941-bc1b1f91a962?ixlib=rb-1.2.1&ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&auto=format&fit=crop&w=1267&q=80",
destFolder: "D:\\images",
destFilename: "2.jpg"
};
const data3 = {
id: 3,
url: "https://images.unsplash.com/photo-1572299409364-c1f3732d2ef5?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=1350&q=80",
destFolder: "D:\\images",
destFilename: "3.jpg"
};
jobQueue.on('task_failed', (taskId, err) => { console.log(`task ${taskId} - error: `, err.toString()); })
.on("task_finish", (taskId, result, stats) => { console.log(`task ${taskId} - result: ${result}, elapsed time: ${stats.elapsed}`); })
.on('task_accepted', taskId => { console.log(`task: ${taskId} accepted`); })
.on('task_queued', taskId => { console.log(`task: ${taskId} queued`); });;
jobQueue.push(data1);
jobQueue.push(data2);
jobQueue.push(data3);
What am I missing here?
PlkMarudny commented
Even a basic documentation example from "Batch processing" produces the same error...