move slider guis to meshcat
RussTedrake opened this issue · 2 comments
Related to #144 . This is a possible work-around for lack of interaction in deepnote (and spotty support for ipywidgets more generally in the jupyter-like cloud notebook ecosystem).
- Can use
set_control
, e.g. viathis.set_control("test", this.mycallback, 50, 0, 100, 0.1);
in the javascript to add a slider to the ui. -
set_control
messages are also parsed in the javascript, it's easy enough to pass them through the zmqserver. - Demonstrate that I can send something back to the zmqserver from the callback.
Update: this works:this.set_control("test", '(value) => this.connection.send("slider=" + value)', 50, 0, 100, 0.1);
- zmqserver should forward to other websocket clients that are not the sender (and store the values?). perhaps I need
update_control
for this? Or a variant ofset_control
that takes only the value. - Design protocol for sending UI updates from zmqserver => python. A few ideas?
1) reuse the existing zmq connection. zmqserver already sends little ack messages back when it receives. I would have to be very careful to not interrupt those handshakes.
2) open a second zmq connection, specifically for the ui callback traffic
3) python could connect to zmqserver as a websocket, and listen only to theupdate_control
messages.
Note that the python client receiving the ui updates need not be the original meschat.Visualizer()
object. I'd be happy to have a separate meshcat.UISubscriber()
type object instead. It might be cleaner.
Another alternative has always been to avoid the zmqserver. I have a working version of meshcat visualizer in python that relies only on ipython kernel comms as the transport. That works great on local machines, but not on colab / deepnote (both of which use a custom kernel implementation).
But perhaps I should consider a visualizer that hosts the websocket connection directly (as zmqserver does now). This would be instead of kernel comms, would have the advantage that it would work outside of jupyter, but would still cut out the zmq layer. This feels like perhaps the best solution. There are still many decisions to be made (do I support multiple clients? do I store the scene tree on the server and allow a new client to get the history, or act more like drake visualizer and require the viewer to connect before any important message is sent?).
Here's a burn-down list for this version
- Spike test running zmqserver in the main jupyter notebook (not as subprocess). Verify I can still connect.
- Spike test the WebSocketVisualizer case in python, taking the easy answer to all of the interesting questions (single client, no memory).
- WebSocketVisualizer should get instantiated once (at the top of the notebook), and we pass a raw pointer or owned Pointer to the meshcat visualizer system. Perhaps it can even be a direct stand-in for the current visualizer. This is important so that users don't have to re-open their visualizer for each simulation.
- Spike test a C++ version of the WebSocketVisualizer.
I've decided to run with the Meshcat in C++ solution for this. Closing this since I've moved the discussion to RobotLocomotion/drake#13038