Gazebosim Web Backend Server

Development and Code style

See https://github.com/golang/go/wiki/CodeReviewComments

The main() and the package init() functions are in application.go. If you are starting to read this code, probably you will want to start from there.

Install

Go version 1.14 or above.

Download the server code

git clone https://github.com/gazebo-web/gazebosim-web-backend

Build and download dependencies
```
cd gazebosim-web-backend
go build
```
Install mysql:

NOTE: Install a version greater than v5.6.4. In the servers, we are currently using MySQL v5.7.21
```
sudo apt-get install default-mysql-server
```
The installer will ask you to create a root password for mysql.

Create the database and a user in mysql. Replace 'newuser' with your username and 'password' with your new password:

 # Xenial
 mysql -u root -p
 # Bionic requires sudo
 sudo mysql -u root -p

CREATE DATABASE gazebo;

Also create a separate database to use with tests:

CREATE DATABASE gazebo_test;

CREATE USER 'newuser'@'localhost' IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON gazebo.* TO 'newuser'@'localhost';

GRANT ALL PRIVILEGES ON gazebo_test.* TO 'newuser'@'localhost';

FLUSH PRIVILEGES;

exit

Create a .env file in the root of the gazebosim-web-backend directory with the following environment variables.

export GZ_DB_USERNAME=<YOUR_MYSQL_USERNAME>
export GZ_DB_PASSWORD=<YOUR_MYSQL_PASSWORD>
export GZ_DB_ADDRESS=localhost:3306
export GZ_DB_NAME=gazebo
export GZ_DB_MAX_OPEN_CONNECTIONS=44

# This is needed when updating library versions
export GZ_VERSION_PASSWORD=<THE_GAZEBO_VERSION_PASSWORD>

# These are needed only for deployment
export AWS_ACCESS_KEY_ID=<YOUR_AWS_ACCESS_KEY>
export AWS_SECRET_ACCESS_KEY=<YOUR_AWS_SECRET_KEY>
export AWS_INSTANCE_NAME_PREFIX=<AWS_INSTANCE_PREFIX>
export AWS_REGION=us-east-1

# This is for authentication
export AUTH0_RSA256_PUBLIC_KEY=<AUTH0_PUBLIC_KEY>

export GZ_HTTP_PORT=8001
export GZ_SSL_PORT=4431
export GZ_LOGGER_LOG_STDOUT=true

Run the server

./gazebosim-web-backend

Try the libs route using

curl http://localhost:8000/1.0/libs

Test

Create a Test JWT token (this is needed for tests to pass OK -- go test)

TL;DR: Just copy and paste the following env vars in your system (.env)

 # Test RSA256 Private key WITHOUT the -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY-----
 # It is used by token-generator to generate the Test JWT Token
 export TOKEN_GENERATOR_PRIVATE_RSA256_KEY=MIICWwIBAAKBgQDdlatRjRjogo3WojgGHFHYLugdUWAY9iR3fy4arWNA1Cos8kVw33cJibXr8bvwUAUparCwlvdbH6dvEOfou0/gCFQsHUfQrSDv+MuSUMAe8jzKE4qW+jK+xQU9a03GUnKHkkle+Q0pX/g6jXZ7r1/xAK5Do2kQ+X5xK9cipRgEKwIDAQABAoGAD+onAtVye4ic7VR7V50DF9bOnwRwNXrARcDhq9LWNRrRGElESYYTQ6EbatXS3MCyjjX2eMhu/aF5YhXBwkppwxg+EOmXeh+MzL7Zh284OuPbkglAaGhV9bb6/5CpuGb1esyPbYW+Ty2PC0GSZfIXkXs76jXAu9TOBvD0ybc2YlkCQQDywg2R/7t3Q2OE2+yo382CLJdrlSLVROWKwb4tb2PjhY4XAwV8d1vy0RenxTB+K5Mu57uVSTHtrMK0GAtFr833AkEA6avx20OHo61Yela/4k5kQDtjEf1N0LfI+BcWZtxsS3jDM3i1Hp0KSu5rsCPb8acJo5RO26gGVrfAsDcIXKC+bQJAZZ2XIpsitLyPpuiMOvBbzPavd4gY6Z8KWrfYzJoI/Q9FuBo6rKwl4BFoToD7WIUS+hpkagwWiz+6zLoX1dbOZwJACmH5fSSjAkLRi54PKJ8TFUeOP15h9sQzydI8zJU+upvDEKZsZc/UhT/SySDOxQ4G/523Y0sz/OZtSWcol/UMgQJALesy++GdvoIDLfJX5GBQpuFgFenRiRDabxrE9MNUZ2aPFaFp+DyAe+b4nDwuJaW2LURbr8AEZga7oQj0uYxcYw==

 # JWT Token generated by the token-generator program using the above Test RSA keys
 # This token does not expire.
 export GZ_TEST_JWT=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ0ZXN0LXVzZXItaWRlbnRpdHkifQ.iV59-kBkZ86XKKsph8fxEeyxDiswY1zvPGi4977cHbbDEkMA3Y3t_zzmwU4JEmjbTeToQZ_qFNJGGNufK2guLy0SAicwjDmv-3dHDfJUH5x1vfi1fZFnmX_b8215BNbCBZU0T2a9DEFypxAQCQyiAQDE9gS8anFLHHlbcWdJdGw

 # A Test RSA256 Public key, without the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----.
 # It is used to override the AUTH0_RSA256_PUBLIC_KEY when tests are run.
 export TEST_RSA256_PUBLIC_KEY=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDdlatRjRjogo3WojgGHFHYLugdUWAY9iR3fy4arWNA1KoS8kVw33cJibXr8bvwUAUparCwlvdbH6dvEOfou0/gCFQsHUfQrSDv+MuSUMAe8jzKE4qW+jK+xQU9a03GUnKHkkle+Q0pX/g6jXZ7r1/xAK5Do2kQ+X5xK9cipRgEKwIDAQAB

In summary, in order to make go test work with JWT you will need to set the following env vars:

TOKEN_GENERATOR_PRIVATE_RSA256_KEY
TEST_RSA256_PUBLIC_KEY
GZ_TEST_JWT

Run the test suite

First, make sure you have all the required env variables set. Below you can see a list of required env vars. But there could be other required vars too. To be completely sure, see the Env Variables section.

export GZ_DB_USERNAME=<DB username>

export GZ_DB_PASSWORD=<DB password>

export GZ_DB_ADDRESS=<DB IP and port. Eg: localhost:3306>

export GZ_DB_NAME=gazebo

export GZ_DB_MAX_OPEN_CONNS=66

export GZ_TEST_JWT=<JWT token>

export TEST_RSA256_PUBLIC_KEY=< RSA256 public key without the '-----BEGIN CERTIFICATE-----' and '-----END CERTIFICATE-----'>
Note: TEST_RSA256_PUBLIC_KEY must be able to decode and validate the GZ_TEST_JWT test token.

export GZ_DB_LOG=false

export GZ_LOGGER_LOG_STDOUT=true

Then, run all tests:

go test github.com/gazebo-web/gazebosim-web-backend

Run the backend server

First, make sure to set the AUTH0_RSA256_PUBLIC_KEY environment variable with the Auth0 RSA256 public key. This env var will be used by the backend to decode and validate any received Auth0 JWT tokens. Note: You can get this key from: https://osrf.auth0.com/.well-known/jwks.json (or from your own auth0 user). Open that url in the browser and copy the value of the x5c field.
```
$GOPATH/bin/ign-webserver
```

Environment Variables

You may want to create an .env file to define environment vars. Remember to add it to .gitignore. This .env will be automatically loaded by the server.

Linter

Get the linter

cd ~/go_ws

curl -L https://git.io/vp6lP | sh

Run the linter

./bin/gometalinter $(go list github.com/gazebo-web/gazebosim-web-backend/...)

Note you can create this bash script:

#!/bin/bash
curl -L https://git.io/vp6lP | sh -s -- -b $GOPATH/bin
$GOPATH/bin/gometalinter $(go list github.com/gazebo-web/gazebosim-web-backend/...)

Deployment

The staging branch in this repository is used to deploy this website to https://staging-api.gazebosim.org. The production branch in this repository is used to deploy this website to https://api.gazebosim.org.

Github actions will automatically deploy staging on push. The production branch will only deploy when an authorized user approves the deployment on the Github Actions UI.

There is no rule about how a release should be made. A person with sufficient access can choose between direct commits or pull requests.

Development

Debugging inside docker container

If you ever need to debug the application as if it were running in AWS or the pipelines, you need to do it from inside its docker containter. To do that:

Most ideas taken from here: Mysql and Docker https://docs.docker.com/samples/library/mysql/#-via-docker-stack-deploy-or-docker-compose

First create the docker image for the ign-webserver server. docker build ign-webserver . Write down its image ID.
Then run a dockerized mysql database. docker run --name my-mysql -e MYSQL_ROOT_PASSWORD=<desired-root-pwd> -d mysql:5.7.21 This will create a mysql docker container with an empty mysql in it.
Then you need to connect to that mysql container and run some commands: docker exec -it my-mysql bash. From inside the container, connect to mysql using the client (eg. mysql -u root -p) and create databases gazebo and gazebo_test. eg: create database gazebo_test;.
Run the ign-webserver docker container and link it to the database. docker run --name ign-webserver --rm --link my-mysql:mysql -ti <ign-webserver-image-id> /bin/bash. This will open a new terminal inside the server container.
Then from inside the server container you will need to set the Env Var that points to the linked docker mysql. eg. export GZ_DB_ADDRESS="172.17.0.2:3306"

After that you can run the server (from inside the container) invoking the ign-webserver command, or run tests by doing go test.