SystemFiles/backuply

BUG: Cannot handle backups of individual files larger than 2GB

Closed this issue · 2 comments

SystemFiles commented

This is likely due to a buffer limit in node being imposed.

I believe this is the culprit

private async _getFileData(files: string[]): Promise<[FileData[], Error]> {
		try {
			const fileData: FileData[] = []
			for (const file of files) {
				const buffer = await readFile(file)
				fileData.push({
					fullPath: file,
					byteLength: buffer.byteLength,
					md5sum: MD5(buffer.toString()),
					deleted: false
				})
			}
			return [ fileData, null ]
		} catch (err) {
			return [ null, err ]
		}
	}

What happens is that the await readFile(file) attempts to load more than 2GB of data into memory (the buffer) and when this happens the limitation is imposed. Obviously I did not mean to be loading large files into memory ... I will likely need to find an alternative means of calculating the MD5 checksum and byteLength(size) for creating the file data for large files.

SystemFiles commented

Additionally, the line md5sum: MD5(buffer.toString()) will fail for large files since the toString() will generate a string from the buffer that could potentially be larger than 0x1fffffe8 characters...