[Discussion] WebSocket Benchmarks and useful graphs / charts
Closed this issue · 5 comments
Hi @mtrudel
I didn't find the discussion section open for this repo, so I am leaving this here.
I feel like there is a need to load test WebSocket connections on Phoenix + Bandit, similar to what was done quite a while ago, using Phoenix + Cowboy (https://www.phoenixframework.org/blog/the-road-to-2-million-websocket-connections)
I feel like there is need of such graphs / charts related to WebSocket, and possibly HTTP too, which I believe helps to understand the performance of Bandit better. There could be a blog post, or some sort of small documentary on load testing Phoenix + Bandit, and to see how many WebSocket connections can be achieved from a single node (as the one in the above link).
Agreed! We have very basic websocket perf testing in our benchmarking suite (you can see the output at e.g.: near the bottom of https://github.com/mtrudel/bandit/actions/runs/7838502587), but they're really intended for us to avoid performance regressions on PRs, not as 'absolute' benchmarks. They also expire from GitHub actions after a few months so they're not great to link to.
I did a run of the WS suite against Bandit and Cowboy's branch heads; you can see it here. It's not ideal (most of the charts only relate to HTTP/1 suites and don't even populate for WS), but it's a start.
If you're curious, the implementation of the benchmarking suite is over here
@mtrudel
Yes, I have seen those GitHub actions benchmarking. Well a suggestion would be that, you can maybe combine all of those benchmarks into a Single GitHub Action, which outputs the result artifacts, but not just in a plain CSV fashion, but also as a static website or something, and it deploys that to GitHub pages. Then it would be a good way to link to it, and put in some extras such as graphs and charts.
Yes, I have seen those GitHub actions benchmarking. Well a suggestion would be that, you can maybe combine all of those benchmarks into a Single GitHub Action, which outputs the result artifacts, but not just in a plain CSV fashion, but also as a static website or something, and it deploys that to GitHub pages. Then it would be a good way to link to it, and put in some extras such as graphs and charts.
Sounds great; a PR for this would be very welcome! I personally am not planning to spend any meaningful amounts of time on measurement or project PR for a while (I try to rotate through all of the aspects of the project in turn, and perf just got a big chunk of my time around this time last year).
If you want to take this on, I'd be more than happy to accept the work on a separate PR. In the meantime, I'm going to close this issue for project hygiene reasons. Thanks for the issue!
@mtrudel Sure, no problem. Just so we don't pollute the Issues with discussion, maybe you can open up the discussions section for this repository, so all related talks can be made there.
Yes, I have seen those GitHub actions benchmarking. Well a suggestion would be that, you can maybe combine all of those benchmarks into a Single GitHub Action, which outputs the result artifacts, but not just in a plain CSV fashion, but also as a static website or something, and it deploys that to GitHub pages. Then it would be a good way to link to it, and put in some extras such as graphs and charts.
Sounds great; a PR for this would be very welcome! I personally am not planning to spend any meaningful amounts of time on measurement or project PR for a while (I try to rotate through all of the aspects of the project in turn, and perf just got a big chunk of my time around this time last year).
If you want to take this on, I'd be more than happy to accept the work on a separate PR. In the meantime, I'm going to close this issue for project hygiene reasons. Thanks for the issue!
I have worked on it here: #300