tus/tus-js-client

docs: Add semantics to fileReader

nh2 opened this issue · 1 comments

nh2 commented

The docs at https://github.com/tus/tus-js-client/blob/beea5983cc79d6d61b65d150cef8809114873661/docs/api.md#filereader show the type signatures, but they don't explain what the values mean.

interface FileReader {
  openFile(input: any, chunkSize: number): Promise<FileSource>
}

interface FileSource {
  size: number
  slice(start: number, end: number): Promise<SliceResult>
  close()
}

interface SliceResult {
  // Platform-specific data type which must be usable by the HTTP stack as a body.
  value: any
  done: boolean
}

For example:

  1. What's the meaning of done: boolean, to what value should one set it? The code at

    tus-js-client/lib/upload.js

    Lines 784 to 791 in beea598

    return this._source.slice(start, end).then(({ value, done }) => {
    // If the upload length is deferred, the upload size was not specified during
    // upload creation. So, if the file reader is done reading, we know the total
    // upload size and can tell the tus server.
    if (this.options.uploadLengthDeferred && done) {
    this._size = this._offset + (value && value.size ? value.size : 0)
    req.setHeader('Upload-Length', this._size)
    }

    suggests it's to indicate the end of the stream when uploadLengthDeferred is used. So I guess there are 2 possible cases:
    a. uploadLengthDeferred = true, FileSource.size = Infinity -- in that case, probably my slice() function must also be prepared to receive end = Infinity, and I should set done = true exactly when I know that the streaming is now finished.
    b. uploadLengthDeferred = false, in that case FileSource.size must be finite, my slice will only be called with finite end, and it is irrelevant what I set done to because the value is not read.
  2. Is there any other contract over what the paremeters in slice(start, end) will be? For example:
    a. Is end the exclusive boundary, like in Blob.slice()?
    b. Will it be that end - start <= chunkSize (the chunkSize that was passed to openFile())?

One can guess answers from the current code, but since this is an interface, it would be nice to know what invariants will hold in the future.

Good point, thanks for bringing this up. The documentation could definitely be improved here.

2. a. Is end the exclusive boundary, like in Blob.slice()?

Yes, see

// The `end` option for createReadStream is treated inclusively
// (see https://nodejs.org/api/fs.html#fs_fs_createreadstream_path_options).
// However, the Buffer#slice(start, end) and also our Source#slice(start, end)
// method treat the end range exclusively, so we have to subtract 1.
// This prevents an off-by-one error when reporting upload progress.
end: offset + end - 1,

2. b. Will it be that end - start <= chunkSize (the chunkSize that was passed to openFile())?

If end is finite, yes. But be aware that end might also be Infinity.

  1. What's the meaning of done: boolean, to what value should one set it?

done indicates if subsequent calls to slice will not return any data because the stream has ended. So done can be set in the final slice call where the final data chunk was returned:

{
  done: true,
  value: [some_data]
}

Or done can be set as the only property if you notice that no more data is available:

{
  done: true,
  value: null
}

Note that in this case, tus-js-client might have to send an empty PATCH request to indicate the end of the upload if you used uploadLengthDeferred.

I am not entirely sure if done is only used if uploadLengthDeferred is enabled, but I encourage every implementation to handle it properly because we cannot guarantee that this might not change in the future.

I will make sure to add all these points to the documentation soon.