diamondio/better-queue

batchSize bigger than 1 => "TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined"

Opened this issue · 1 comments

When I try to use queue in a batch mode, any batchSize > 1 causes tasks to fail with an error:

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined

The setup is minimal:

const downloadFile = async (fileUrl, downloadFolder, fileName) => {

    const localFilePath = path.resolve(__dirname, downloadFolder, fileName);
    try {
        const response = await axios({
            method: "GET",
            url: fileUrl,
            responseType: "stream",
        });

        await response.data.pipe(fs.createWriteStream(localFilePath));
        return fileName;
    } catch (err) {
        throw (err);
    }
};

const cb = (arg) => { console.log(arg); };

const processData = (data, cb) => {
    downloadFile(data.url, data.destFolder, data.destFilename).then((result) =>
        cb(null, `file ${result} downloaded`)).catch((err) => {
            // console.log(err);
            cb(err, null);
        });
};

const jobQueue = new Queue(processData, { batchSize: 2, concurrent: 1 });

const data1 = {
    id: 1,
    url: "https://images.unsplash.com/photo-1616548479562-d5a581c064f4?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=2125&q=80",
    destFolder: "D:\\images",
    destFilename: "1.jpg"
};
// this one should fail
const data2 = {
    id: 2,
    url: "https://images.unsplash.co/photo-1550389941-bc1b1f91a962?ixlib=rb-1.2.1&ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&auto=format&fit=crop&w=1267&q=80",
    destFolder: "D:\\images",
    destFilename: "2.jpg"
};
const data3 = {
    id: 3,
    url: "https://images.unsplash.com/photo-1572299409364-c1f3732d2ef5?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=1350&q=80",
    destFolder: "D:\\images",
    destFilename: "3.jpg"
};

jobQueue.on('task_failed', (taskId, err) => { console.log(`task ${taskId} - error: `, err.toString()); })
    .on("task_finish", (taskId, result, stats) => { console.log(`task ${taskId} - result: ${result}, elapsed time: ${stats.elapsed}`); })
    .on('task_accepted', taskId => { console.log(`task: ${taskId} accepted`); })
    .on('task_queued', taskId => { console.log(`task: ${taskId} queued`); });;

jobQueue.push(data1);
jobQueue.push(data2);
jobQueue.push(data3);

What am I missing here?

Even a basic documentation example from "Batch processing" produces the same error...