Enables a person (the operator) to remotely teleoperate a Stretch RE1 (the robot) through a recent Chrome/Chromium web browser on an Android mobile phone, laptop, or desktop. This code is a heavily modified fork of the original interface from Hello Robot.
- Multiple command modes: incremental and continuous
- Beam-style click-to-drive
- Map UI for navigation
- Supports add-on camera feeds
- Written in Typescript
- Vanilla, component-based code
WARNING: This prototype code has been useful to the community, but is not well tested. There are also security issues, especially if you use the default credentials. Use this code at your own risk.
This web interface works via Web Real-Time Communication (WebRTC). Code runs in a browser on the robot, in a browser on the operator's device (e.g., a mobile phone), and on a server. This is analogous to the robot and the operator video conferencing with one another, although they communicate via realtime data in addition to audio and video. The class that manages the WebRTC connection, webrtcconnection.ts, is the same on both the robot and the operator side. By using web browsers, the robot and the operator make use of well-tested high-performance implementations of WebRTC. This symmetry also simplifies development, since a developer can use the same browser-based developer tools on both sides of the communication. The robot's browser and the operator's browser first login to the server, which helps connect them by providing a signaling message channel (signaling_sockets.js) as well as by serving the interface code.
The robot’s browser uses rosbridge to connect with ROS on the robot. Rosbridge translates JSON from the robot’s browser into ROS communications and vice versa. The JavaScript code used by the robot’s browser to connect with ROS can be found in robot.ts under the robot directory, which holds files made available to the robot's browser.
With puppeteer the robot can automatically launch and login to its browser. For example, start_robot_browser.js uses puppeteer to launch the robot's browser and login.
While the robot’s browser has access to most of the robot via ROS, the operator’s browser can only access the robot indirectly through the robot’s browser. The robotic commands available to the operator’s browser can be found in commands.ts under the shared directory, which holds files available to both the operator's browser and the robot's browser. The operator's browser also has access to files in the operator directory.
In the example below, the server runs on the robot. In a production environment, you would use an external server, instead of the robot, to handle things like connecting robots and operators behind firewalls. In a later section, we provide an example of an external server that uses Amazon Lightsail. When used on a production server with proper certificates, this code uses HTTPS and avoids scary messages.
The web server uses the Express web framework with Pug templates. The server provides a WebRTC signaling service using socket.io. It uses Redis to store sessions.
passport provides authentication for the robot and the operator. mongoose and a MongoDB database store credentials for robots and operators. The stretch_web_interface repository comes with default MongoDB content found at ./mongodb/ for testing behind a firewall. These default contents come with multiple robot and operator accounts. Make sure not to use these default database contents on a deployed system!
By default, the code uses a free STUN server provided by Google. The Amazon Lightsail example below uses coturn as a STUN and TURN server.
These installation instructions describe how to install both the server and relevant ROS code on the onboard computer of a Stretch RE1 robot. This is suitable for use on a trusted and secure local area network (LAN) behind a strong firewall.
The web interface depends on stretch_ros, which is used to control the robot. You should first make sure it is up-to-date and working properly on the robot.
Clone the stretch_web_interface repository to ~/catkin_ws/src/ on the robot.
cd ~/catkin_ws/src/
git clone https://github.com/hcrlab/stretch_web_interface.git
Run catkin_make.
cd ~/catkin_ws/
catkin_make
rospack profile
Run the installation script.
cd ~/catkin_ws/src/stretch_web_interface/bash_scripts/
sudo ./web_interface_installation.sh
WARNING: The script uninstalls tornado using pip to avoid a rosbridge websocket immediate disconnection issue. This could break other software on your robot.
Browser features like camera and microphone access require that the page be running in an SSL context, so we need certificates in order to serve the interface and to enable SSL for the rosbridge websocket (see the launch files). For development, we suggest you use mkcert to manage a set of certificates across the development machines and your robot. These certificates are only valid on machines that are configured to trust them, so you should still use Let's Encrypt if you need to deploy the interface to users who won't have this configuration.
To trust the certificates we've provided (which cover forky.hcrlab.cs.washington.edu, our robot's hostname within the University of Washington, as well as localhost and variants):
roscd stretch_web_interface/certificates && CAROOT=`pwd` mkcert --install
To make your own certificates for your preferred development hostnames, follow mkcert's instructions. We suggest something like mkcert forky.hcrlab.cs.washington.edu forky.local forky.dev localhost 127.0.0.1 0.0.0.0 ::1. Then copy the "Certificate Authority" that mkcert created on your machine under ~/.local/share/mkcert to the other machines, then use the CAROOT environment variable (as in the previous invocation) to specify the authority and install the certificate you generated.
Note: Some browsers may allow you to access the interface without having added the custom certificate authority to the truststore by clicking through some scary prompts.
When running on a trusted and secure local area network (LAN) behind a strong firewall, you can use the following insecure method to more conveniently start the system.
First, make sure the robot is calibrated. For example you can run the following command.
stretch_robot_home.py
Next, in a terminal, run the following command to start ROS. This will start ROS nodes on the robot for the D435i camera, the driver for Stretch RE1, and rosbridge. Rosbridge connects JavaScript running in the robot's browser to ROS using JSON.
roslaunch stretch_web_interface web_interface.launch
In another terminal, run the following command to start the web server on the robot, launch the robot's browser, and log the robot into the browser. The convenience script calls start_robot_browser.js, which uses puppeteer to log the robot into its browser.
roscd stretch_web_interface/bash_scripts/
./start_web_server_and_robot_browser.sh
Typically, this script can be exited with Ctrl+C and then restarted without issue.
WARNING: start_robot_browser.js contains the default robot credentials in plain text! This is only appropriate for simple testing on a local network behind a firewall. The username and password are public on the Internet, so this is not secure! Deployment would require new credentials and security measures.
You will now login to a browser as the operator and connect to the robot. You can use a Chrome browser on a recent Android mobile phone or a recent Chrome/Chromium browser on a laptop or desktop.
Open the browser goto the robot’s IP address. You can use ifconfig on the robot to determine its IP address.
Select "Advanced" and then click on "Proceed to localhost (unsafe)".
Click on "Login" and use the following username and password.
username:
o1
password
xXTgfdH8
WARNING: This is a default operator account provided for simple testing. Since this username and password are public on the Internet, this is not secure. You should only use this behind a firewall during development and testing. Deployment would require new credentials and security measures.
You should now see a screen like the following. Click on "no robot connected" and select the robot "r1" to connect to it.
You should now see video from the robot on your mobile phone or other device. Click in the designated regions to command the robot to move. You can also click on "Drive", "Arm" down, "Arm" up, "Hand" and "Look" to move different joints on the robot.
The following steps describe how to manually start the web server and the robot's browser on the robot, instead of using the convenience script described above.
First, make sure the robot is calibrated. For example you can run the following command.
stretch_robot_home.py
Next, in a terminal, run the following command to start the ROS side of things. This will start ROS nodes on the robot for the D435i camera, the driver for Stretch RE1, and rosbridge. Rosbridge connects JavaScript running in the robot's browser to ROS using JSON.
roslaunch stretch_web_interface web_interface.launch
In another terminal, run the following command to start the web server on the robot.
roscd stretch_web_interface/bash_scripts/
./start_desktop_dev_env.sh
Open a Chromium browser on the robot and go to localhost. Select "Advanced" and then click on "Proceed to localhost (unsafe)".
Click on "Login" and use the following username and password.
username:
r1
password
NQUeUb98
WARNING: This is a default robot account provided for simple testing. Since this username and password are public on the Internet, this is not secure. You should only use this behind a firewall during development and testing. Deployment would require new credentials and security measures.
You should now see video from the robot's camera in the browser window.
Please see the instructions above.
The server for the web interface typically runs on the robot's onboard computer or on a remote machine connected to the Internet.
Credentials for robots and operators are stored on the server using MongoDB.
On the server, you can view and edit the credentials using mongodb-compass, which is installed by default. First, use the following command in a terminal to start the application.
mongodb-compass
Next, use "Connect to Host" by typing localhost in the Hostname area at the top of the window and then clicking the green "CONNECT" button at the bottom right of the window. This should show you various databases. The node-auth database holds the web interface credentials.
Clicking on node-auth will show a collection named users.
Clicking on users will show the current credentials.
If you've only used the default development credentials in this repository, you should see entries for the following: three robots with the usernames r1, r2, and r3; three operators with the usernames o1, o2, and o3; and an administrator with the username admin. Each entry consists of encrypted password information (i.e., salt and hash), a username, a name, a role, a date, and a Boolean indicating whether or not the user has been approved. Without approval, the user should be denied access. The role indicates whether the entry is for a robot or an operator. You can click on the image below to see what this should look like.
First, start the server. Next, go to the web page and click register. Now enter a username and a password. This process creates a new user entry in MongoDB.
You can now follow the instructions for viewing credentials above to view the new account you just created. In order for this account to function, you will need to edit the role to be operator or robot and edit approved to be true. You can do this by clicking on the elements with mongodb-compass.
Prior to testing anything on the Internet, you should delete all of the default credentials. The default credentials are solely for development on a secure local network behind a firewall.
On the server, you can backup credentials using a command like the following in a terminal. You should change ./ to match the directory into which you want to saved the backup directory.
mongodump --db node-auth --out ./
You can restore backed up credentials using the following command in a terminal. You'll need to change ./ to match the directory that holds the backup directory.
mongorestore -d node-auth ./node-auth/user.bson
Running the server on the robot is useful when the robot and the operator are both on the same local area network (LAN). For example, a person with disabilities might operate the robot in their home, or you might be developing new teleoperation code. In these situations, the robot, the operator's browser, and the server should all be behind a strong firewall, reducing security concerns.
Running the server on a remote machine can be useful when the robot and the operator are on separate LANs connected by the Internet. This can enable a person to operate the robot from across the world. In this section, we'll provide an example of setting up the server to run on an Amazon Lightsail instance. This is not a hardened server and is only intended to serve as a helpful example. It likely has significant security shortcomings and is very much a prototype. Use at your own risk.
One of the challenges for remote teleoperation is that browsers on different LANs can have difficulty connecting with one another. Peer-to-peer communication may not be achievable due to firewalls and other methods used to help secure networks. For example, home networks, university networks, and corporate networks can all have complex configurations that interfere with peer-to-peer communication. Running the server on a remote machine connected to the Internet helps the robot's browser and the operator's browser connect to one another using standard methods developed for WebRTC video conferencing over the Internet. The server performs a variety of roles, including the following: restricting access to authorized robots and operators; helping operators select from available robots; WebRTC signaling, Session Traversal Utilities for Network Address Translation (STUN), and Traversal Using Relays around Network Address Translation (TURN). Notably, when direct peer-to-peer connectivity fails, TURN relays video, audio, and data between the robot's browser and the operator's browser. Relaying data is robust to networking challenges, but can incur charges due to data usage.
This section describes the steps we used to create an Amazon Lightsail instance that runs the server for the web interface.
- Obtain a domain name to use for your server.
- Create a new Amazon Lightsail instance.
- We used an OS only instance with Ubuntu 20.04, 512 MB RAM, 1 vCPU, 20 GB SSD.
- Create a static IP address and attach it to your instance.
- Connect your new instance to SSH, so that you can access it.
- A command like the following can then be used to login to your instance:
ssh -i /path/to/private-key.pem username@public-ip-address.
- A command like the following can then be used to login to your instance:
- While logged into your instance.
- Run
sudo apt-get updateto avoid installation issues. - Install helpful packages.
sudo apt install emacssudo apt install net-tools
- Configure Git.
git config --global user.name "FIRST_NAME LAST_NAME"git config --global user.email "MY_NAME@example.com"
- Clone this GitHub repository.
cdmkdir reposgit clone https://github.com/hello-robot/stretch_web_interface.git
- Use certbot from Let's Encrypt to obtain certificates so that your server can use Hypertext Transfer Protocol Secure (HTTPS). HTTPS is required to fully utilize WebRTC.
- You will need to first connect your domain name to the static IP address used by your instance.
- Follow certbot installation instructions for Ubuntu 20.04.
- Run the teleoperation server installation script
cd ~/repos/stretch_web_interface/bash_scripts/./web_server_installation.sh
- Run
- Initialize the database with secure credentials for at least one robot and one operator. For example, you can do the following.
- Create and export credentials from MongoDB by running a server on your robot.
- Use
scpto copy the exported credentials database from your robot's computer to your Lightsail instance by running a command likescp -ri ./LightsailDefaultKey-us-east-2.pem ./mongodb_credentials ubuntu@public-ip-address:./on your robot's computer. - On your Lightsail instance, restore the credentials database with a command like
mongorestore -d node-auth ./mongodb_credentials/node-auth/users.bson
- While logged into your instance.
- Edit the coturn configuration file.
- Find your instance's private IP address by running
ifconfig -aand looking at theinetIP address. - Confirm your instance's public IP address and your domain name by running
ping YOUR-DOMAIN-NAMEand looking at the IP address. - Add the following lines at appropriate locations in
/etc/turnserver.conf.listening-ip=PRIVATE-IP-ADDRESSrelay-ip=PRIVATE-IP-ADDRESSexternal-ip=PUBLIC-IP-ADDRESSVerboselt-cred-mechpkey=/etc/letsencrypt/live/YOUR-DOMAIN-NAME/privkey.pemcert=/etc/letsencrypt/live/YOUR-DOMAIN-NAME/cert.pemno-multicast-peerssecure-stunmobilityrealm=YOUR-DOMAIN-NAME
- Find your instance's private IP address by running
- Create TURN server accounts and credentials.
- Create an administrator account.
sudo turnadmin -A -u ADMIN-NAME -p ADMIN-PASSWORD
- Create a TURN user. In the next step, you will add these credentials to
./stretch_web_interface/shared/send_recv_av.js.sudo turnadmin -a -u TURN-USER-NAME -r YOUR-DOMAIN-NAME -p TURN-USER-PASSWORD
- Open
./stretch_web_interface/shared/send_recv_av.jsin an editor.- Comment out the free STUN server.
- Uncomment the STUN and TURN servers and fill in the values using your domain name and the credentials you just created (i.e., YOUR-DOMAIN-NAME, TURN-USER-NAME, and TURN-USER-PASSWORD).
- The relevant code will look similar to
var pcConfig = { iceServers: [ {urls: "stun:YOUR-DOMAIN-NAME", username "TURN-USER-NAME", credentials: "TURN-USER-PASSWORD}, {urls: "turn:YOUR-DOMAIN-NAME", username "TURN-USER-NAME", credentials: "TURN-USER-PASSWORD}]};
- The relevant code will look similar to
- Create an administrator account.
- Edit the coturn configuration file.
- Open the following ports for your Amazon Lightsail instance. These are standard ports for HTTPS, STUN, and TURN.
HTTPS TCP 443Custom TCP 3478Custom TCP 5349Custom UDP 3478Custom UDP 5349
- Reboot your instance.
- Login to your instance and run the following commands to start the server.
cd ~/repos/stretch_web_interface/bash_scripts/./start_server_production_env.sh
- Your server should now be running and you can test it by taking the following steps.
- Turn on and calibrate your robot.
- Use your robot's Chromium browser to visit your server's domain and login with the robot's credentials that you created.
- Open a Chrome or Chromium browser of your own on another computer or recent Android phone. Visit your server's domain and login with the operator credentials you created. If you want to test communication between distinct networks, you could turn off your phone's Wi-Fi and then either use your phone or tether to your phone to connect from the mobile phone network to your robot. Please note that this has only been tested with the Chrome browser on recent Android phones.
- Once you've logged in as an operator, you should be able to select your robot from the drop down list and begin controlling it.
- After trying it out, be sure to shutdown your Amazon Lightsail instance. It is not hardened and likely has security vulnerabilities. It would be risky to leave it on over a significant length of time.
The following license applies to the contents of this directory written by Maya Cakmak, Kavi Dey, Anna Garverick, Charles C. Kemp, Vinitha Ranganeni, Joe Sluis, Nick Walker, Brian Yao, authors associated with Hello Robot Inc., and authors associated with the University of Washington (the "Contents"). This software is intended for use with Stretch ® mobile manipulators produced and sold by Hello Robot ®.
Copyright 2023 Maya Cakmak, Kavi Dey, Vinitha Ranganeni, Joe Sluis, Nick Walker, Brian Yao, and Hello Robot Inc.
The Contents are licensed under the Apache License, Version 2.0 (the "License"). You may not use the Contents except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, the Contents are distributed under the License are distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.