johang/btfs

How it downloads - documentation request

Tomas-M opened this issue · 5 comments

Tomas-M commented

Hello,
would you please describe in docs or github readme how the torrent files are downloaded on read() operation?

For example, if I mount a torrent file with a movie, and then I use mplayer to play the movie, is it going to be downloaded from beginning to the end? Or is it downloaded on background as like any other torrent client would download it, meaning parts of the beginning of the file will be missing long time after the download starts, so the file will be basically unplayable until it is completely downloaded?

Is the file downloaded on background? Or is the data downloaded only on demand upon read() operation?

I tried to mount a torrent with some movie, and it appears to be downloading all the time even if no files are read from the filesystem. A brief description how it operates would be really useful. Thank you!

Ivshti commented

AFAIK this works the following way:
(authors: if I'm wrong, please correct me!)

  1. initializes libtorrent for the given torrent
  2. on read(), the pieces the read() is waiting for will be prioritized; as soon as they arrive, the result will be returned from read() (so no, the whole file does not have to be downloaded)
  3. it is possible that this operation also selects all the pieces from the given file, which explains why it continues to be downloaded

Also, depending on your needs, you should also check out https://github.com/mafintosh/torrent-stream, which is kind of like libtorrent but allows much better piece prioritization for exactly that kind of purpose; there's also a fuse library here: https://github.com/mafintosh/torrent-mount ; even though both are implemented in JS, they achieve performance comparable to libtorrent

johang commented

BTFS maintains a sliding window of pieces (https://en.wikipedia.org/wiki/Sliding_window_protocol). When a program reads a file, this window will be moved to that offset. The window will be moved forward when pieces finish. The window will be moved forward even if read() is not called. That's why it will continue to download.

mtbar131 commented

Can you please explain how you start/stop downloading of pieces? I can see that you set the file_priority to 0 in init function but this stops libtorrent from downloading anything. Then how do you make it start downloading again when a read request comes? Or when a read request is fulfilled how do you stop it from downloading any other pieces?
Also, when I tried to mount a mp4 file using btfs VLC was not able to play anything. What might be the issue?

johang commented

Yes, file_priority is set to 0 for all files in the setup function. Later, when a read happens it will raise priority for the pieces in the requested range. See this call chain: btfs_read(), Read::read(), Read::jump(). Read::jump() moves the sliding window to the beginning of the range.

mtbar131 commented

I am trying to achieve similar thing but when I set file priority to 0 then even after setting individual piece priority to 7 libtorrent does not download anything. My code simply keeps waiting for download of piece to finish. I guess, libtorrent doesn't even connect to peers. Is there any specific setting that needs to be done? I have tried setting auto_managed to false but it didn't help.