filesystem.writeBinaryFile() appears to not work on Array Buffers above a certain size

Question

filesystem.writeBinaryFile() appears to not work on Array Buffers above a certain size

dimitrinossar opened this issue a year ago · 7 comments

Hi, I'm writing a small project to convert audio files and I've found what I think might be an issue with the writeBinaryFile API. Array Buffers that are above a certain size seem to be choking the program but it won't throw any obvious errors.

const output = ffmpeg.FS('readFile', download)
await filesystem.writeBinaryFile(`${saveLocation}/${download}`, output)

Logging the output will always return a valid uint8array regardless of the file format and lossy audio files (such as mp3) will write to the location fine and continue on with the program's control flow but larger lossless audio files (flac, wav) seem to hang endlessly on the await but importantly refuse to throw an NE_FS_FILWRER error.

To give an idea of the sizes I'm dealing with, my limited test involved an mp3 around 11MB, a flac around 32MB, and a wav around 58mb.

Answer 1 · 2024-02-22T14:36:01.000Z

Hey, I am interested to work on this issue, Please assign to me

Answer 2 · 2024-02-22T18:18:30.000Z

I have checked it and It doesn't gives me this error

Answer 3 · 2024-02-23T18:06:15.000Z

Hi, I'm writing a small project to convert audio files and I've found what I think might be an issue with the writeBinaryFile API. Array Buffers that are above a certain size seem to be choking the program but it won't throw any obvious errors.
const output = ffmpeg.FS('readFile', download)
await filesystem.writeBinaryFile(`${saveLocation}/${download}`, output)
Logging the output will always return a valid uint8array regardless of the file format and lossy audio files (such as mp3) will write to the location fine and continue on with the program's control flow but larger lossless audio files (flac, wav) seem to hang endlessly on the await but importantly refuse to throw an NE_FS_FILWRER error.

To give an idea of the sizes I'm dealing with, my limited test involved an mp3 around 11MB, a flac around 32MB, and a wav around 58mb.

hey @dimitrinossar, are you still facing this issue? could please provide some more information like the operating system you are using? and what is the size limit after which the function stops working?

Answer 4 · 2024-02-26T16:05:38.000Z

@shalithasuranga It seems like if the file size is big then this kind of issue occurring
My approach is to divide the data into small chunks and then send it because Most modern WebSocket implementations should be able to handle 1MB of data in a single message without any issues but in this case, the size is higher than 1 MB so we need to slice it and then send it.

Answer 5 · 2024-02-29T17:23:36.000Z

hii i can slove the issue please assign it to me

Answer 6 · 2024-03-01T06:23:24.000Z

Hello, @dimitrinossar, looks like we need to implement file write streams, similar to openFile() binary read mode. Will you be able to select appendBinaryFile() as a workaround by splitting files from the app?

Note: I've removed gsoc label from this issue since it looks like a feature request to the framework core.

Thanks 🎉

Answer 7 · 2024-03-15T20:18:18.000Z

I ran into this issue a few months back. Downloaded files somewhere between 20-30mb and greater started causing this issue. I was testing using ubuntu 22.04.