Roadmap
Sean-Der opened this issue Β· 75 comments
This is a covering ticket for the Pion WebRTC roadmap. If you have any requests feel free to leave them here/vote on others.
Planet Strike -- 1.0.0 (2018-06-20)
- SRTP (in Go)
- ICE-lite (We only respond to binding requests, requires host candidates. No STUN or TURN)
- DTLS (In C, using OpenSSL)
- API that allows users to receive media that closely matches the Javascript WebRTC API
- API that allows users to send media that closely matches the Javascript WebRTC API
Aliens of Gold -- 1.1.0 (2018-09-00)
- DataChannels
- STUN
- Full ICE (currently ice-lite only)
- Bundling
Quest for the Sigil -- 1.2.0 (2018-12-00)
- Port DTLS to Go from cgo (currently using OpenSSL)
- Mobile Support
- Transport Refactor
Spear of Destiny -- 2.0.0 (2019-04-00)
- ORTC
- Unified Plan
- DataChannel improvements (configurable reliability)
- Logging and ICE debugging
- Experimental QUIC support
- Experimental WebAssembly (WASM) support
- API Cleanup to better match the WebRTC Spec.
- Reliability improvements to SCTP and therefore Data Channels.
- ICE Regular Nomination
Beyond Heretic -- 2.1.0 (2019-08-00)
- TURN
- Trickle ICE
- mDNS Candidates
The HUNT Begins - 2.2.0 (2019-11-00)
- SCTP Performance Improvements (16x in some cases!)
- Extend SettingEngine to support SFU use cases
- TCP TURN Support
- PCM Support
- Add/Remove tracks at runtime
- IVF Reader to enable saving to disk
Future Shock - 3.0.0 (2020-12-14)
- ICE Restarts
- ICE TCP Candidates
- Remove non-Trickle ICE. Provide utility that emulates old behavior.
- SRTP AEAD_AES_128_GCM. Significant performance improvement in media crypto!
- Simulcast.
- Improved Media API so user doesn't need to managed PayloadType and SSRC
- Interceptor API that allows users to implement NACK, FEC and Congestion Control via Public API
Plutonia - 3.1.0 (2021-08-01)
- Transport Wide Congestion Control Interceptor, better Congestion Control then Sender+Receiver Reports
- H265 Packetizer
- FireFox Simulcast Support
- Sender/Recevier Report interceptors, can be used in pion/webrtc or outside
- UDPMux, Multiple PeerConnections can be served via the same UDP Port
System Shock 3.2.0 (2023-04-25)
- Congestion Control Interceptor
- AV1 Support
- Simulcast Sender
- Stats Interceptor
4.0.0 (2024-10-01)
- Active ICE TCP
- JitterBuffer Interceptor
No date picked
- Serialize/Deserialize PeerConnection API
- DTLS Restart
- Performance/Allocation testing to GitHub Actions
- Auto Change log generation/Remove manual process of releasing
- Automate fuzz testing, submit Pion to oss-fuzz
- H264 Interceptor (Analyze and attempt to fix common issues like no SPS/PPS with every IDR)
- Harden security (fuzzing and other automated tooling)
- TinyGo support. Pion for IoT devices
These things are ordered in the importance we have gotten from feedback. If you are building something and need a feature sooner please comment and we can move things around!
It seems it is a webrtc golang version. Any plan on SFU media server?
Hey @notedit
Currently no, we may build an SFU using pions-WebRTC in the future though. You are right though, pions-WebRTC is a Golang version of libwebrtc (instead we don't use any libraries)
The goal is that people will be able to just do one go build and not have to worry about anything else, and other cool things like that!
If you do want to build an SFU would love to help! Can add features/debug things as needed
Thanks, I may try SFU media server based on your lib. Great work.
Pretty excited about this. Personally I'm waiting for DataChannel support.
Many p2p projects (libp2p/ipfs, openbazaar, etc) rely on datachannels to create DHTs of peers, and end up having segregated networks between browser-based WebRTC peers vs native UDP-based peers, only being able to talk to each other over bridges that support both (usually NodeJS).
Having a robust Go-native WebRTC implementation with DataChannel support would be a huge milestone for all of these projects. Most of these projects are written in Go, too.
libp2p/go-libp2p#188 for example.
@shazow Hey thanks for checking us out :)
Just moved it up! I will work on datachannels right after I finish sending video. I have had multiple people reach out to me about wanting that the most, so seems the right choice.
I will try to reach out to any projects that are looking for datachannels. It would be great if they had input on what features they needed most, so I could get it good enough as soon as possible.
Hey @Sean-Der, very very keen to see Data Channels implemented in this library as I'm aiming to build a real-time browser-based game in Golang.
I want Data Channels / UDP to avoid hitches and non-responsiveness when building games with an authoritative server model.
(Source if you're interested: https://new.gafferongames.com/post/why_cant_i_send_udp_packets_from_a_browser/ )
As of right now, there's no "off the shelf" libraries I can use to get here. The only other library that seemed reasonable was keroserene/go-webrtc, but unfortunately it has no pre-built Windows binaries so it's a non-starter for me as I dev' on Windows.
If donations will help get this done quicker, I'm more than happy to contribute some money!
All. Did you also see that the libp2p library also now has QUIC.
It's a pure golang alternative to webrtc data channels.
Huge
@gedw99 QUIC is not an alternative to WebRTC Data Channels. Browsers can't use QUIC to establish peer-to-peer connections with each other.
@silbinarywolf Hey!
Thanks for checking us out :) That's really cool, I am excited to hear about all the use cases like this.
What I was thinking about doing was starting a bountysource (I personally don't need the money/will not incentive me anymore) but was hoping to incentive other contributors. I also might not be presenting work in a way that others can take on.
I think if others could make money AND I could say this could be really good for a resume/career etc... I could get some good Golang devs to help out. But not exactly sure where to start with that.
@shazow since your into libp2p repo then their QUIC implementation might be of interest.
https://github.com/libp2p/go-libp2p-quic-transport
And yes QUIC is not supported in all browsers, but it is supported on all mobiles and desktops because it has no dependencies at all. NOt even kernel level for the networking.
All i am saying is that its quite feasible that pions can support both webrtc and quic through an interface. Not saying it has to be this way but just leaving options on the table.
Hi @Sean-Der. Thanks for your effort. Just wondering, is it possible to build a live steaming / broadcasting system with this library? I want to record video with Navigator.getUserMedia() then stream (via webrtc) to server then distribute to clients.
@kafein Hey!
Yes you can (and pretty easily!) would you mind posting to our mailing list happy to answer any questions there.
We have got quite a lot of questions in this issue, I am going to hide everything for now. I would love to discuss anything on the mailing list though! I just don't want new users to be overwhelmed by unrelated conversations.
Thanks everyone.
Good news!
1.0.0 has officially landed you can see the announcement here look forward to hearing what people think!
I am excited to announce that @nicolai86 has landed STUN support for pion-WebRTC in master! We are still adding tests/improving things but it works.
DataChannels is coming along nicely. I would estimate ~2 weeks, me and @kc5nra are making good progress. We have to fully implement an SCTP parser, but after that should be smooth sailing!
@backkem is also making HUGE progress with his SDP improvements. Currently things in that area are MVP at best, but after his work lands you should expect an API that looks pretty much exactly like WebRTC in the browser.
DataChannels has landed!
You can now use WebRTC DataChannels in master, there are still plenty of things to improve but it works :) if you get a chance try it out and report any bugs! It isn't production ready yet, but getting there. There is an example application here
Hi! Awesome project! What would be necessary for mobile support?
Shouldn't it work out of the box, since it is Go only?
We really need mobile functionality and want to cooperate with that.
I saw that it is not part of the road map, but if we are willing to help, what would be a good starting point?
Best,
Chris
Hi @chrisprobst, we definitely want to support your use-case. Can you elaborate a bit more wat exactly you want to accomplish? E.g.: connect with mobile browsers, compile with go-mobile or ...? Please also consider joining us in the #pion channel of the gophers slack. We're really active on there.
Hi @backkem , nice to hear from you!
Yes, gomobile is our target. We are StriveCDN and building a Go-based SDK for P2P live streaming optimization. Connection would be between SDK (Android and iOS) <-> Web Browsers on Desktop.
I'll join you guys on slack, perfect!
We'll move the go-mobile conversation to #64. I'll hide the conversation here to keep this topic clean.
pion-WebRTC is officially 100% Go, our OpenSSL dependency is gone! I am really excited to announce we have a beta version here I would really appreciate if people could test it. If no issues comes up I plan on merging this around December 1st.
I also need help getting pions/dtls accepted into x/net. You can see the issue here. If you know anyone working on the Go team, please reach out to them! I wouldn't like pion to own the DTLS package long term. Lots of other projects would benefit from it, and putting it into x/net would benefit everyone. It would also benefit greatly from a security review.
Lastly (and most importantly) a big thank you to @backkem without him this wouldn't have happened. He worked very hard on both design+code, so if you are benefiting from this change... send him a thank you!
1.2.0 has officially landed you can see the announcement here look forward to hearing what people think!
Thank you to everyone who has contributed. The team gets so much joy every time we get feedback (good or bad!) and seeing every ones projects makes it all worth it. Enjoy your holidays!
2.0.0 is coming along nicely!
With this release we are going to be making some breaking changes that we have documented here
While building Pion WebRTC I made some mistakes, and taking this chance to fix them! We will also be including ORTC and QUIC support with this release. After this lands we will start working on TURN support (and this will not break the public API)
If anyone has any feedback I would love to hear from you via GitHub issues or on Slack. Thanks again for using Pion.
Looks like 2.0.0 is now here
Hey @rebootcode
It is available! We are still working on some SCTP improvements before we do a big announce, but there will be zero public API changes going forward.
No plan for gccοΌ pcc?
v2.0.* has officially landed and stabilized! Multiple large production users have migrated to it and we fixed all the bugs we found along the way.
This was a really big release, you can see the full release notes here here are some exciting parts though.
- Unified Plan and Plan-B support
- QUIC Support
- WebAssembly Support
- ICE Regular Nomination (better support for NATs)
- Massive SCTP performance and reliability improvements
- API Cleanup to better match the WebRTC Spec
We are also starting work on v2.1.* which so far will include the following. If you have a feature that you think is important we would love to hear!
- TURN Support
- User configurable JitterBuffer
- Re-negotiation (allow multiple Offer/Answer)
Don't forget to publish the v2.0 release here: https://github.com/pion/webrtc/releases/ π
TURN support has landed in master! Pion WebRTC can now connect to a TURN server and maintain an allocation (TURN session).
If you are using an older version of Pion WebRTC it is still possible to use TURN! The remote WebRTC session has to maintain the allocation however. Pion WebRTC can accept relay candidates sent from the other side, but before couldn't create them itself.
@porjo thanks for calling that out, sorry I completely missed the message :( will publish these!
This's an amazing project!
I am playing with some examples, it seems no QoS, any plan for that?
Hey @gibson-luo
The plan is to have a user configurable jitterbuffer (so you can decide what latency you are willing to tolerate and get stats around jitter/bandwidth/latency)
For stuff like REMB I plan on emitting those stats to the user (and you can adjust the encoder accordingly) if you are interested in this I would love your feedback/opinions on Slack
Sounds good.
Yeah, I'm interested in this part, I will try to implement parts of them by myself, just for learning.
Looking forward to it, thank you.
mDNS support has landed!
Currently we only parse them, but don't emit them ourselves. We plan on making this an option that can be controlled via the SettingEngine turning them on by default has caused breakage for Chromium users and we don't want to do the same.
We have a dedicated Contributing page now. I would love to see more people getting involved. I would love to hear feedback if you are interested in getting involved, but we aren't providing the tools or information to make it possible.
@enobufs is working hard on vnet support which is an exciting new feature. You will be able to simulate network topologies (and their conditions) via code. This means you can test how your application will perform in the real world without having to test in the field.
I am going through and triaging issues. I don't plan on adding any new features (just bugfixes) until we release the next dot release.
Thanks for using Pion!
v2.1.0 of Pion WebRTC has arrived!
This release features full TURN support, mDNS candidates, vnet and Trickle ICE! You can read the full release notes here
If you have questions about Pion WebRTC (or even just WebRTC in general!) we would love to have you in our Slack channel!
We also are always looking for more contributors, it is a great way to learn and we try to make it as easy as possible. You can read about that here here
I haven't planned the next release yet, I will announce the features here when we have a rough idea. Our roadmap is really driven by user need, so if you have something you think is important please reach out!
We are closing in on releasing v2.2.0 of Pion WebRTC! Here are the major features we expect from this release. We also have had lots of great bug fixes along the way.
- SCTP Performance Improvements (16x in some cases!)
- Extend SettingEngine to support SFU use cases
- TCP TURN Support
- PCM Support
- Add/Remove tracks at runtime
- IVF Reader to enable saving to disk
Updating to this release is suggested for everyone, we aim to have zero breaking changes with lots of improvements. Also starting with this release we are going to explore how we can better support business users.
We want to explore the idea of providing paid support. We also may be starting a low bandwidth mailing list where we announce upcoming Pion changes and let users who support the project drive the roadmap. We also want to share general WebRTC information that might help anyone who is depending on WebRTC.
@Sean-Der Great, we are looking forward to the SCTP performance improvements!
Can you already suggest when the release will be available? In the meantime, is there a tagged branch with this improvement :-) ?
@chrisprobst Yes! I just pushed the tag v2.1.6 that has the SCTP changes.
pion/sctp@fc51cd9 is the pion/sctp commit and pion/sctp#62 is the tracking issue.
If you get a chance drop @enobufs a thank you! He is always looking for data points that can help make it better, he does some really amazing work :)
Related to the
Explore providing easy bindings for other languages (Python, Java... ?)
My personal suggestion here would be to introduce gRPC (https://grpc.io/) due to numerous reasons. Some of them would be:
- Bi-directional streaming support
- Support for 12 languages officially with numerous additional unofficial support (such as Elixir)
- Easy to get started
- A well mature project that we can count for LTS
@Sean-Der I'm really looking forward to RtpEngine, when are you planning to release itοΌ
Pion WebRTC now allows you to Add/Remove tracks at anytime! You can call AddTrack/RemoveTrack on either side (Pion or Browser) and it should work :)
I have landed this in master, and will wait a week for it to stabilize. The next release of Pion WebRTC is going to be all media API improvements. Supporting multiple browsers is needlessly hard right now, and I want to fix that.
We also are going to work on bringing way more maturity to the getStats API. Pion WebRTC has seen lots of usage in load testing, bench marking, analytics and QA. If you are using Pion WebRTC for any of these we would love to hear your feedback about what would make it better for you!
@helisong427 RtpEngine is something I hope to do before April! If it is blocking something you are trying to build I would love to do it sooner.
I am mostly driven by things people need, if you are able to get involved/explain your exact uses cases that would really be helpful! I am in Slack if you ever want to talk real-time :)
@Sean-Der just know that this is a really awesome project, and your work is greatly appreciated.
Love this feature:
- Add/Remove tracks at runtime
given your roadmap it looks it might be a good idea for some bufferbloat.net folk to get involved in this?
@dtaht Hey sorry I missed this, I would love the input from the bufferbloat community! I just went and read about it now, and looks super interesting.
I haven't had a chance to implement congestion control yet, but when I do will be great to get advice from others. With TWCC we will get a bunch of stats, and then I was planning on making it pluggable.
rmcat provides a bunch of algorithms, I just need to go and implement them I guess :)
TinyGo support. Pion for IoT devices
Hey @TaoZhangHao
I would love to see this supported! If you are interested you should work on it! Happy to answer questions/give advice I am available on Slack our or mailing list!
I am so excited to announce that Interceptors have landed! pion/webrtc will now have pluggable RTP/RTCP processors. This means you can conditionally enable/disable and tweak things like Error Correction and Congestion Control.
By default we will enable NACK and Receiver Reports. You can disable them, and configure the RTP pipeline however you want though.
We are working on stabilizing everything. I plan on tagging v3.0.0 within the week.
Sean, Any plan on Native mobile SDKs (included iOS android)
Hey @BenjaminX
Do you want to run Pion on the device, or are you connect iOS/Android to a Pion server? Both are possible today.
sfu-ws shows how you can connect iOS/Android to a Pion server using Flutter.
You can use gomobile like in go-yuv2webrtc. c-data-channels shows how to use FFI without gomobile. You can build a shared object and then call that from your Andriod/iOS code.
Hey @BenjaminX
Do you want to run Pion on the device, or are you connect iOS/Android to a Pion server? Both are possible today.
sfu-ws shows how you can connect iOS/Android to a Pion server using Flutter.
You can use gomobile like in go-yuv2webrtc. c-data-channels shows how to use FFI without gomobile. You can build a shared object and then call that from your Andriod/iOS code.
Thanks a lot, sean. Have you checked this project https://github.com/flutter-webrtc , it should be integrated into Pion
I am kind of curious if any level of ecn support could be added. There are several competing proposals, notably SCE (which I favor) vs L4S (which is currently in the IETF). I've
Secondly, some fiddling with galene recently showed that it is still impossible to seperate voice and video tuples, which could be a very good congestion control mechanism in a fair queued environment.
Hey @dtaht sorry it took so long to respond. My backlog grows daily....
It is possible to run audio/video on separate tuples with WebRTC, Pion doesn't support it unfortunately. I would love your input on what we are doing with Congestion Control! I have no plans/ideas to interact with ECN currently just because I don't know enough.
If you have a chance I would love to chat in Slack if you don't want to deal with it I totally understand :)
Hi Sean,
we have a webrtc/xmpp-jingle based unified communication service for which we build sdk which allow our users to build bots, applications, mobile apps, web apps..
I am trying to evaluate viable solutions for a multiplatform bot sdk either based on python, or on go.
So using pion and go-xmpp, I built a proof of concept sample that handles the signalization, trickle, and it works fine, passes the firewalls.. But the thing I miss in pion that might be a show stopper (I still have to demo and try to convince my hierarchy that we should go this way) for us is support for telephony events, which I am afraid is probably not on the top of your todo list.
I have no idea how complex it is to implement, maybe we can do it ourself ?
any hints on that ? what we would like is be able to build bots able to react to telephony events as described in https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Using_DTMF .
Thanks in advance :)
It's funny, I remember when we decided to skip implicating DTMF in the past.
I'm assuming you only want to send DTMF? The WebRTC spec doesn't include an API for the receiving side. (Note that if you do need the receiver, we can likely figure out a way to fit it into the codebase regardless.)
If you want to contribute support you need to add at least:
- DTMF API
- The logic for signaling the audio/telephone-event codec.
- rfc4733: the actual audio/telephone-event codec. As referenced in rfc7874. This would likely take the form of a DTMFTransport that wraps the underlying RTPTransport.
These are not the most complex protocols in the stack but it will take a bit of work to implement.
All major features have landed for v3.1.0
Transport Wide Congestion control is especially exciting. With Sender/Receiver reports ramping up to the ceiling bitrate could take almost a minute, now it is instant.
https://github.com/pion/webrtc/milestones/3.1.0 is tracking what is left in the release. Testing is much appreciated. We should have a final release tagged in the next week.
Did you ever get anyone working on "my" issues? :) I'm not sure what you mean by "transport wide congestion control" (link?), but for me, it's ramping down to the right bitrate is what I care about most.
@dtaht yes!
TWCC is just the feedback messages so the remote can adjust their estimate. In pion/interceptor#25 (comment) we have plans to write the code to make our own estimates.
We can't use ECN today for Congestion Control, but https://datatracker.ietf.org/doc/html/rfc8888 (TWCC + 1) does! So we don't have anything yet, but in the future we will :)
Pion WebRTC now provides a Congestion Controller in tree! It is an implementation of the Google Congestion Controller. You can see an example of its usage here to see the actual code see pion/interceptor. All credit goes to @mengelbart who designed and implemented it.
The Congestion Controller is designed in a way that allows you to choose between multiple algorithms or write your own. We provide an interface and helpers so you only need to solve the important parts. A NADA implementation (written by @kevmo314) in the near future will be merged into pion/interceptor.
I am now going to start preparing a v3.2.0 release. I am going to try and close easy bugs and triage issues. If anyone is interested I have made a list here. Even if you can only help with one it will have a huge impact.
WHIP WHEP support? Or just an example?
WHIP WHEP support? Or just an example?
WHIP/WHEP can be implemented with your own HTTP request or server right now, here's a server example: https://github.com/kevmo314/tinywhip/blob/main/cmd/main.go and a client example: https://github.com/ggarber/whip-go/
Happy to help clarify those examples, it seems like a good idea to put a simpler example under examples/ too.
Please add support for MKI (Master Key Identifier). This consists of two parts:
- first one is to support sending and receiving SRTP/SRTCP packets with single configured MKI. This will be useful for basic case when MKI has to be used;
- second part is to allow use of multiple MKIs, what is needed for re-keying. Sender side seems to be quite easy (just replace one pair key+MKI with another). Receiving side is more complicated as code has to handle multiple keys at the same time.
BTW, DTLS-SRTP contains srtp_mki field, you could add support for it too.
One more thing, please also add support for non-zero key derivation rate for SRTP.
I think we should add windows testing
Add support for RFC 7675 - Session Traversal Utilities for NAT (STUN) Usage for Consent Freshness. WHEP specification requires it for media servers:
A Media Server terminating a session MUST follow the procedures in RFC7675 section 5.2 for immediate revocation of consent.
Pion #WebRTC v4.0.0 is here. I am so grateful for the work everyone has put into it. https://github.com/pion/webrtc/wiki/Release-WebRTC@v4.0.0
- Distinct SSRCes are fully supported for retransmissions
- DTLS Close (immediate hangups)
- SCTP Zero Checksum
- Censorship Circumvention
- WHIP/WHEP examples
So much left to do still. Things can still better like AV1/H265 support, FlexFEC and more WebRTC Stats. Here is an opportunity to learn interesting things and work on impactful code! You can make an amazing career out of it, I am here to help.
Shoot me a message sean @ pion.ly or join Slack. Open Source had a enormous impact on my life, it can do the same for you.