slow with very large files
michaelkirk opened this issue · 3 comments
michaelkirk commented
I'm using koa-range via local-web-server to serve up small slices (e.g. 12kb) of a large (12GB) file. The response time is very slow - on the order of 30+ seconds.
I'm not at all familiar with node, but I'm somewhat suspicious of this line, which seems to read and discard from the very beginning of the file.
rawBody = rawBody.pipe(slice(start, end + 1));
That rawBody
seems to come from this line in koa-send
:
ctx.body = fs.createReadStream(path)
It might be better if we can find a way to stream only the requested portion of the data , rather than streaming the entire stream through the slice
function.
e.g. somewhere in koa-send:
ctx.body = fs.createReadStream(path, { start, end });
michaelkirk commented
I hacked a POC and verified that the responses are now super fast.
diff -r node_modules/koa-range/index.js hacks-node-modules/koa-range/index.js
25a26,31
> // avoid multi ranges
> var firstRange = ranges[0];
> var start = firstRange[0];
> var end = firstRange[1];
> ctx.req.foo_range = { start, end };
>
33c39,40
< var first = ranges[0];
---
>
>
37,42c44
< // avoid multi ranges
< var firstRange = ranges[0];
< var start = firstRange[0];
< var end = firstRange[1];
<
< if (!Buffer.isBuffer(rawBody)) {
---
> if (!Buffer.isBuffer(rawBody)) {
45c47,48
< rawBody = rawBody.pipe(slice(start, end + 1));
---
> // redundant, since it's been sliced in koa-send
> // rawBody = rawBody.pipe(slice(start, end + 1));
diff -r node_modules/koa-send/index.js hacks-node-modules/koa-send/index.js
153c153,161
< ctx.body = fs.createReadStream(path)
---
> if (ctx.req.foo_range != undefined) {
> let start = ctx.req.foo_range.start;
> let end = ctx.req.foo_range.end;
> console.log(`range requested: ${start}-${end}`);
> ctx.body = fs.createReadStream(path, { start, end })
> } else {
> console.log("no range requested");
> ctx.body = fs.createReadStream(path)
> }
masx200 commented
I also encountered the same problem