downloadFile files allocate huge memories
ueokande opened this issue · 0 comments
ueokande commented
Package
- @kintone/rest-api-client 3.1.11
Why
The FileClinet downloads blobs into memory. This occurs high memory allocation depends on file size.
I created kintone records which have 3 attachments, each attachment have 1GiB file size. The following code allocate over 6GiB memory size.
// index.js
const { KintoneRestAPIClient } = require("@kintone/rest-api-client");
const client = new KintoneRestAPIClient({
baseUrl: "https://my-domain.kintone.com",
auth: {
username: process.env.KINTONE_USERNAME,
password: process.env.KINTONE_PASSWORD,
},
});
(async () => {
const { record } = await client.record.getRecord({ app: 1, id: 2 });
const blobs = await Promise.all(
record["Attachment"].value.map(value => client.file.downloadFile({
fileKey: value.fileKey,
}))
)
for (const blob of blobs) {
console.log("Downloaded blob: ", blob.length);
}
for (const [key, value] of Object.entries(process.memoryUsage())) {
console.log(`${key}: ${Math.round(value / 1024 / 1024 * 100) / 100} MiB`)
}
})();
$ node index.js
Downloaded blob: 1073741824
Downloaded blob: 1073741824
Downloaded blob: 1073741824
rss: 6628.86 MiB
heapTotal: 287.4 MiB
heapUsed: 203.38 MiB
external: 5145.33 MiB
arrayBuffers: 5124.03 MiB
What
Well-known libraries and APIs returns a stream rather than buffer array. fetch()
API returns a ReadableStream. It is useful to pipe other process. Of course using streams enable to download into memory.