biigle/user-storage

Try to improve AssembleChunkedFile

Closed this issue · 0 comments

mzur commented

The AssembleChunkedFile job downloads each chunk and appends it to a temporary local file:

$tempFile = fopen($filename, 'w+');
$path = $this->file->request->getPendingPath($this->file->path);
$chunkPaths = [];
for ($i = 0; $i < $this->file->total_chunks; $i++) {
$chunkPath = "{$path}.{$i}";
$chunkPaths[] = $chunkPath;
$stream = $disk->readStream($chunkPath);
stream_copy_to_stream($stream, $tempFile);
}
fseek($tempFile, 0);
$success = $disk->writeStream($path, $tempFile);
fclose($tempFile);

Instead, it may be possible to append to the final storage disk file directly (with readStream() and writeStream()). Find out if this works and if yes, update the job.

Background: There was a case with a huge file where the locally available storage space was not sufficient. If everything is done directly in the storage disk, this would no longer be a problem.