Add WebRTC support to popular torrent clients
feross opened this issue · 67 comments
Meta issue to track progress on getting WebRTC support into popular torrent clients.
👍
@dcposch has done some exploration of getting webrtc support into libtorrent. It looks like it'll be doable!
@feross seems to me that this should be the most important outstanding issue by far as it is really the only blocker for webtorrent gaining widespread use in the browser and fulfilling the original promise of the project.
@feross @mafintosh One other extremely popular torrent client that isn't listed here would be peerflix, as it still sees ridiculous usage (although hard to quantify) via the various popcorn-time forks. If peerflix were augmented to support webrtc peers, that could potentially allow a popcorn-time in the browser use case to function very well. Product Hunters seem to agree that this would be a great development.
It's also pretty clear that given the synergy between torrent-stream/peerflix and webtorrent in terms of environment and shared dependencies, making peerflix peers hybrid wouldn't be nearly as difficult as integrating webrtc support into the other, non-js-based torrent clients.
Interesting...
peerflix peers hybrid wouldn't be nearly as difficult as integrating webrtc support into the other, non-js-based torrent clients
Agreed. @jhiesey & i checked out how hard it would be to add webtorrent support to one of the native torrent libraries, eg libtorrent.
It looks like the biggest hurdle is that there's no libdatachannel or anything like that, no small library with a clean API for using WebRTC data channels. instead, there are two full WebRTC implementations:
- https://github.com/js-platform/libwebrtc looks pretty tightly coupled to the Chromium build process, and is really complex. In fact, as far as i can tell, they literally pasted in much of the Chromium source code, including parts that have nothing to do with WebRTC. See, for example: https://github.com/js-platform/libwebrtc/tree/master/chromium/src/net/ftp
- https://github.com/EricssonResearch/openwebrtc looks smaller and cleaner, but still has a lot of video & audio handling code which we'd have to remove.
Maybe this project could be helpful: https://github.com/xhs/librtcdc
wow, that looks awesome. if i have time later i'll try it out
Sorry if this is a stupid question, but:
Webtorrent's docs/FAQ suggest that browser-based WebTorrent users can only communicate with other browser based clients and "hybrid" clients. "Hybrid" clients can communicate with either pool, bridging the swarms.
Ultimately we don't want bridges, we want one swarm. But right now, what are the "bridges" or "hybrids"? Node-based WebTorrent users? Or something else?
@cathalgarvey there is a node hybrid client : see https://github.com/feross/webtorrent-hybrid (also see webtorrent/webtorrent-hybrid#5 (comment))
@feross
Did you reach to bittorrent inc ? Making webtorrent as extension to Bittorrent protocol would be huge step.
The most important adopters are uTorrent and libtorrent (rtorrent, deluge etc.), rest really doesn't matter.
Another thing, where is documentation of protocol which webtorrent uses with implementation details ?
This is a must for adoption and I do not see any links in github repo or on webtorrent.io. You need to standardize webtorrent or you will see no adoption at all.
If someone would like to implement it in C/C++ then where he should look ? What if he is not familiar with javascript ?
In my opinion this is the main reason why you do not see any adoption besides people using your own lib.
@devlo completely agreed; I know feross is aware of this, but there just hasn't been anyone who's stepped up and been able to make this integration happen. The piece that I'd recommend starting from would be communicating in C from libtorrent to https://github.com/feross/webtorrent-hybrid via webrtc probably using one of the C-based webrtc libraries listed above.
See libtorrent#223 for some continued libtorrent discussion.
https://wiki.vuze.com/w/WebTorrent it seems vuze has support now !
It seems to me that documenting the protocol adaptations/extensions would be the best way to encourage the developers of other clients to add support to their codebases. Ideally, I suppose this would come in the form of a BEP but even a draft specification would likely be welcomed by many.
Add in this issue the link for libtorrent issue on github #223
Duplicated, sorry!
Someone created a ticket for Transmission. It may need some details though.
transmission/transmission#47
If this were done for libtorrent, it would work for deluge, and qbittorrent, the two most popular open source bittorrent clients. Once that's done, you instantly got a lot more seeders.
@feross I'm planning on assisting adding this into libtorrent, with a c++ webRTC library, but could you help point me to which files in this codebase show the webRTC communication?
Great @dessalines! Cannot wait to see that coming!
If I can give you some pointers:
- You will have to connect to the websocket tracker in order to make WebRTC signalling. Basically you will have to add WebRTC offers on announce and manage SDP offers/responses. Here is our implementation: https://github.com/feross/bittorrent-tracker/blob/master/lib/client/websocket-tracker.js
- Once a WebRTC connexion is established, the torrent peer protocol is used, so there is nothing special here. This project uses https://github.com/feross/simple-peer to make the connexion then it becomes a "normal" peer. (Except that the connexion is handled upstream: https://github.com/feross/webtorrent/blob/15ed59a0d2c9d32598ab4c98706e6fd8fa200843/lib/peer.js#L16)
There is a BEP we are working on in #881, it should give you some informations but be aware that some fields have been renamed for the future standard and read the comments (or look at the code).
The document is published at https://github.com/yciabaud/webtorrent/blob/486a94d2b651e7aed86ba27b82e679f5d1d1e700/bep_webrtc.rst
I am very interested in making libtorrent support webtorrent so ask if there is anything I can do for it.
Great thread. Any more intel on updates to popular clients? I have used Vuze but cant seem to find if Delluge and other have recently adopted webRTC or not.
Vuze doesnt seem to be working. I cant find anybody who is using it to ask either. For me its not and I have tried it in every way on every type of device and location I can reasonably try it on.
@garycarlyle Vuze does work for me. I tried today on my mac. Have you use the latest version? The only problem here with Vuze implementation is that it spawn a chromium process on every file ... :(
Sorry I've been slow on helping out with this yall, had a few other projects come up. Jeez, spawning a chromium process for every torrent sounds like a horrible way to do this.
Here's the issue for libtorrent, we need to pull in a C webrtc library, and add a connection class alongside bt and httpseed/web peers.
I would need 5,000 Chromium process' then lol.
And that was only the beta version of my project.
I got it (Vuze) to work on my laptop which is good but still not on my desktop and VPS servers.
Vuze support is practically non-existant even though I bought several copies of the premium.
Not sure why companies screw themselves like that.
Libtorrent seems to be the way to go but personally I know not enough about it so I cant help sadly. Although I will look into it too to see if I can contribute anything.
As it looks like bittornado has no webrtc client. Please test-download any torrent from https://tracker.mxchange.org/ and you will see that it will download with any non-webrtc client but not with webrtc.
Edit: Or is it using libtorrent? Then it is sure that it doesn't support WebRTC. Nope, it is Python.
BitTornado is being used on my server for mass-seeding those files. And oh, it is Python, not C/C++. So no libtorrent there.
FYI, Brave Browser just added WebTorrent support!
Press: https://torrentfreak.com/brave-a-privacy-focused-browser-with-built-in-torrent-streaming-170219/
Press: https://torrentfreak.com/utorrent-will-move-to-the-web-browser-170421/
Does it mean that uTorrent will support WebRTC?
@feross you mentioned back in January you were working on a spec. Has there been any progress on this?
@t0xicDen For me it sounds more like a web interface for the client, similiar to Plex for example.
Hey guys. I'm throwing in RAWRTC here as an alternative to openwebrtc, librtcdc & co. Feel free to ping me if you have any questions about it.
Any news about this?
Seems pretty much dead.
Here is the issue for rtorrent and libtorrent-rakshasa
rakshasa/rtorrent#717
To avoid confusion, we should also use libtorrent-rasterbar instead of just libtorrent.
@feross add on the list
Vuze as a seedbox kind/not works http://hamishcampbell.com/index.php/2018/05/18/testing-peertube-webrtc-seedboxes/
From what I understand, as of today there is no client usable on a headless server right?
webtorrent hybrid
Request to add BiglyBT to the list.
It can be seen as the continuation of Vuze as it's development has stalled.
https://sourceforge.net/p/azureus/mailman/azureus-commitlog/
https://www.biglybt.com/
https://github.com/BiglySoftware/BiglyBT/graphs/contributors
https://en.wikipedia.org/wiki/Vuze (search for BiglyBT)
As for WebRTC support, I'm testing it but it seems to have NAT and/or UPnP issues for me. So I can't confirm yet if it's usable.
edit 2018-06-23: I'm not sure that BiglyBT and Vuze can actually seed with WebRTC. I could barely seed when with WebTorrent desktop it seeds most of the time.
Is this dead?
Is this dead?
No.
And implements webrtc without browser... Its possible?
And this... https://github.com/pion/webrtc and https://github.com/thorst/RTC
Yes, there’s discussion over at arvidn/libtorrent#223 about existing non‑browser WebRTC implementations and their pros and cons.
The most favourable option currently appears to be @lgrahl’s RAWRTC.
Anyone into Go may be able to pick up anacrolix/torrent#138. pion/webrtc should be in good shape to power a Go implementation.
Libtorrent (which is written in C++) is the only one that matters. It powers qbittorrent, deluge, transmission, rtorrent.
rTorrent uses its own BitTorrent implementation, which is also, confusingly, called LibTorrent.
Any update?
I really want to upload my videos to a seedbox and then stream to peertube, bitchute and web players, but it seems odd to have to do it trough a VPS and a freaking browser open. It's just too much bloat because I need to install X and so on.
I don't trust peertube instances to keep hosting.
Really interesting progress on integrating into libtorrent. arvidn/libtorrent#4123
I really hope this is still on-going. I would love to see something like rTorrent adopting WebRTC support. This will be a game changer. Been following this for a very long time but it seems no new clients added WebRTC for years :(.
is there by now any high-performance torrent client which i can use to seed a huge amount of files?
i saw there is some movement at libtorrent but nothing usable until now.
anacrolix/torrent just landed initial WebTorrent support in anacrolix/torrent#393. I guess this marks the first port to another language, Go in this case.
The Wikipedia Article comparing BitTorrent clients is missing WebTorrent clients and also a column on WebTorrent/WebRTC as a supported feature. That would also be a good place to track support, I hope this is on topic here.
libtorrent (arvidn/libtorrent#223, #241) implementation finshed, awaiting review to go/ready
PR arvidn/libtorrent#4123 is merged!
Please add this too: https://github.com/Novik/ruTorrent
seedboxes from seedbox.io use them
@modbender
I think rtorrent is what you are after as ruTorrent is just the php browser interface.
Oh right. Didn't read about it fully, Thank you.
Is there any guide to compile rtorrent with webrtc support?
//sorry for offtopic.
@feross I think bringing WRTC to JSTorrent will be hard for the same reasons why the brave webtorrent extension doesn't have WRTC, also libtorrent has WRTC support, but doesn't support webtorrent yet, it wants to do it tho
Is there any guide to compile rtorrent with webrtc support?
//sorry for offtopic.
I think isn't have any movement to turn hybrid the rtorrent (rakshasa/rtorrent#717)
btw... I commented on @vbakc issue, i liked the TUI of client
I'm not familiar with the Torrent but from what I understood it would be better to focus on Web Seeding with plain HTTP instead of WebRTC.
We already have many files in the existing Torrent network but they just aren't accessible by a browser. The WebRTC allows a browser to seed files but for most users they just need to get a content and they even may want not to seed anything, especially from a mobile phone.
I briefly read the spec BEP 19: WebSeed - HTTP/FTP Seeding in GetRight style and as far I understood the Web Seeding was intended to make a file downloaded by clients via HTTP and then they may seed it over the torrent network as usual.
It would be great if the clients can seed it too over HTTP to increase seeders swarm.
A Torrent tracker just stores part ranges with hashes and list of seeders.
Also this may allow us to use just a router as a web server. Both OpenWrt uhttpd and BusyBox httpd supports the Accept-Range header. Also a router then can use just a simple wget to download torrents. This would be extremely great to download package updates.
I think that for a simplicity a tracker may be replaced with some Dynamic DNS proxy that receives a magnet hash as a subdomain, performs a DHT lookup and returns an IP of a seeder. Then to download a file it should be enough to write a simple shell script with for loop that calls wget and checks hash sum.
P.S. Just found a zsync tool which does the same but without a tracker which is also a very useful.
Please correct me if I wrong because I asked a very strange questions.
I working on a YurtPage homepage engine for small routers and I need to implement a content distribution over Torrent so that routers can handle photo and video hosting.
In the same time the content must be backed up or distributed between routers of friends and followers. So the router must also act as client itself.
UPD here is the dedicated issue for this #1492
With respect, I feel like your proposal, as I understand it, does not reflect the usecases of many Bittorrent and WebTorrent users. Putting aside some of your recommendations for infrastructure changes for a moment, and just examining the sugguestion of moving to advocacy for increased Web seeding in torrenting ecosystems rather than advocacy for WebTorrent support, which I believe is the argument you make in the first part of your message.
The key flaw I see is that Web Seeds are selected at torrent creation, and cannot be changed at any point after. This introduces two key issues:
- This is a central point of failure. If a swarm is dependent on a small selection of webservers, the loss of a few of these will degrade torrent performance, and the loss of all of them could jeporadize or even kill the swarm.
- This undermines the core appeal of BitTorrent/WebTorrent, which is consumers as providers. Using torrents, it is trivially easy, automatic often, for a user who downloads some content to then provide that content (both in terms of disk space / availability over time and in terms of bandwidth) to the swarm. This resiliancy and performance is a key appeal of WebTorrent. WebSeeds remove this relation, as consumers have no way to offer the data the downloaded or any of their bandwidth to the swarm. Of course, they can seed it over the normal BitTorrent protocol or WebTorrent protocol, but this divides the swarm and impairs resiliancy and performance if these two protocols can't co-operate, hence this issue.
On the topic of some of the infrastructure changes you proposed, I don't believe that very many members of bittorrent/webtorrent communities operate using routers as their main avenue for retrieving torrents, instead prefering desktop applications or web applications. Many of these users do not have the technical knowledge to set up dynamic DNS on an ICANN-recognized domain, although to be honest, I may just be misunderstanding this part of your proposal, as I admit I had a little trouble following.
Reading through your usecase, it sounds like you already have an idea of what peers you want to be sharing/receiving files from. It might be a good idea to not use trackers in favor of manually adding peers, which I belive you only need a ip/port pair for. If you would like to use trackers, hosting a tracker on the sharing router might also be worthwhile?
Although full disclaimer, I'm not a WebTorrent developer, so someone else might be more qualified to chip in
@Alch-Emi thank you for your comment. My main idea is to make torrent working on HTTP in addition to mTP. The HTTP protocol can be directly used from a browser. The main advantage would be that instead of a big WebRTC library a torrent client and server can have just a basic HTTP client and server.
The WebSeeds looks like implements a server part only. So probably it's already possible to download a torrent from a browser without WS and WebRTC.
As an advantage of the HTTP is that any existing web server, including smallest one on routers can handle this. And this also allows to make torrent daemon smaller and reuse a HTTP daemon.
theory is nice, practice isnt!
BT is made in a way where data is consumed in chunks, HTTP doesn't like that very much, and currently there's no way of actually control incoming data streams.
I made this: #2192
but to say its... bad would be an understatement, this is partially an issue with how webtorrent handles webseeds, and partially the fault of the browser.
WebRTC libs aren't that big, for example libdatachannel is just 500KB, which isn't that big.
Overall I think this is insanely off-topic as the main idea is to help WebTorrent download from existing peers, rather than create new peers and new implementations to connect to those peers. WebTorrent supports webseeds [but not get-right style]
@ThaUnknown thank you for the explanation. I still didn't fully understood what's wrong with HTTP but I believe you.