/retro-cloud

An expensive and over-engineered approach to storing your ROMs and their metadata.

Primary LanguageShellMIT LicenseMIT

Running ROMs from the cloud

An expensive and over-engineered approach to storing ROMs and their metadata which sets out to answer the question:

Why buy a cheap USB stick when you can use multiple expensive services in the Cloud?

Setup

  1. Install Retro-Cloud on the Raspberry Pi (creates the VM for step 2):

    $ curl -fsSL https://raw.githubusercontent.com/seriema/retro-cloud/master/raspberry-pi/download-and-run.sh | bash
# Or:
$ curl -fsSL https://tiny.cc/retro-cloud-setup | bash

    NOTE! You will be prompted to log into your Azure account. The script pauses with the message:

    WARNING: To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code ABCD12345 to authenticate.

    • In case the setup fails before creating all necessary Azure resources it's easiest to tear them down and try again:
      1. $ cd retro-cloud-setup
      2. $ bash teardown.sh
      3. $ rm -f "$HOME/.retro-cloud.env"
      4. $ bash setup.sh

  2. Install Retro-Cloud on the VM. Alternatives:

    • On the Raspberry Pi:

      $ bash -i setup-vm.sh

    • On the VM. Log into the VM from the RPi with $ bash -i ssh-vm.sh, or any other way you want, and then run:

      $ curl -fsSL https://raw.githubusercontent.com/seriema/retro-cloud/master/virtual-machine/setup.sh | bash
# Or:
$ curl -fsSL https://tiny.cc/retro-cloud-setup-vm | bash

  3. Copy ROMs to Azure File Share. Alternatives:

    • If you already had ROMs on the Raspberry Pi: They're now in roms.bak and can be copied over:

      $ bash -i copy-roms-to-file-share.sh

    • If you have ROMs on a desktop: Use Azure Storage Explorer and copy them to Storage Accounts/[numbers]storage/Files Shares/retro-cloud/RetroPie/roms

  4. Add scraper credentials. Retro-Cloud uses Skyscraper by Lars Muldjord for scraping. It supports adding credentials the scraper modules, which can allow you to scrape using more threads, priority when the source is under heavy load, or access to sources that require credentials. Read more here. Alternatives:

    • On the Raspberry Pi: $ ./add-scraper-credential.sh MODULE USER PASSWORD
    • On the VM: $ ./add-scraper-credential.sh MODULE USER PASSWORD

  5. Scrape for metadata. Alternatives:

    Note: This will take a long time. A test run of 6 platforms with 13k files took 10 hours. EmulationStation must not be running during this time.

    • On the Raspberry Pi: $ bash -i run-scraper.sh
    • On the VM: $ ./run-skyscraper.sh

Development

Architecture

architecture-diagram

File structure

filestructure-diagram

Prerequisites

  • PowerShell 7+
  • Bash 4.4.12+
  • Docker Engine 19+ (Optional, but highly recommended)

Workflow

This project is in large part install scripts running on someone's Raspberry Pi for the first time. To ensure installation works it's crucial to test on a new RetroPie installation each time. Wiping a SD-card after every test can be time consuming, so there is a Docker image to make this convenient. Therefore, there are three main areas of development: test environment, install scripts, and functionality.

  1. The test environment: A best-effort Docker image to represent a newly installed RetroPie installation, as there is no official Docker image for neither Raspbian (the RaspberryPi OS) nor RetroPie. It can run slimmed down as a user's RetroPie, or with some install scripts pre-run and with the source code available inside the container.
  2. The install scripts: Bash and PowerShell scripts that installs all the needed tools and applications to create and connect all Azure resources with the user's RetroPie.

    Tip: Use docker/start.sh while working on the scripts for faster iteration, and start.sh to test run them from scratch.

  3. The functionality: Part of the install scripts, but focusing on the features available after installation. Such as ROM scraping.
  • Development
    • docker/setup.sh sets up environment variables for automation (same as used in CI) in a .env file in the root. Avoid the Azure login prompt by setting up a Service Principle account.

      Tip: Use docker/dev/create-service-principal.ps1 to conveniently create a Service Principle, and store the results in the .env file.

    • docker/build.sh to build a Docker image meant for running locally. The tag is rc:[branch name].
    • docker/test.sh to validate the built Docker image.
    • docker/start.sh to run a throwaway Docker container that:
      • Makes the source code available inside the container as ~/retro-cloud-source, and uses it as the work directory.
      • Caches some install steps as named volumes (i.e. PowerShell).
      • Adds environment variables from the .env file for automation (see docker/setup.sh).
    • docker/publish-arm.sh to build and push an ARM image of the master branch to Docker Hub as seriema/retro-cloud:latest-arm32v7. The AMD image is built by Docker Hub automated builds as seriema/retro-cloud:latest-amd64.
    • lint.sh to lint all .sh files through a Docker image with shellcheck, and check their execution permissions.
    • .env file: Used to store environment variables useful during development. It consists of:
      1. Azure automation
        • AZURE_TENANT_ID
        • AZURE_SERVICE_PRINCIPAL_USER
        • AZURE_SERVICE_PRINCIPAL_SECRET
      2. Retro-Cloud setup (re-use an Azure infrastructure between runs by copying these values from the RPi's $HOME/.retro-cloud.env)
        • RETROCLOUD_VM_IP
        • RETROCLOUD_VM_USER
        • RETROCLOUD_VM_SHARE
        • RETROCLOUD_AZ_RESOURCE_GROUP
      3. CircleCI scripting
        • CIRCLECI_API_USER_TOKEN
  • Testing install scripts as a user
    • start.sh to run a throwaway Docker container using the latest release image.
    • To test a specific branch:
      • branch=[the branch you want to test], e.g. branch=upgrade-powershell
      • curl -fOL "https://raw.githubusercontent.com/seriema/retro-cloud/${branch}/raspberry-pi/download-and-run.sh"
      • bash download-and-run.sh "$branch"
      • rm download-and-run.sh
    • To validate that the image is correct:
      • docker/compose/run_tests.sh
  • Continous Integration with CircleCI
    • .circleci/build-all-commits.sh to queue a build for every commit between current branch and develop. Meant to be used before creating a PR or merging to develop so that each commit is validated.

Notes on Windows

Some of the Bash scripts above don't work on Windows (Git Bash) so there are some PowerShell equivalents. They aren't guaranteed to be as powerfull as the Bash scripts.

  • Development
    • docker/start.ps1
    • docker/test.ps1

    Note: Requires File Sharing to be enabled. See the Docker documentation.

    • .sh files are required to have execute permission. To do this in git bash for Windows:

      $ git update-index --chmod=+x [file]

Release checklist

Use a real Raspberry Pi with Docker to test with.

  1. Checkout develop

  2. Create a release branch release/1.X.Y

  3. Push, to start CI tests

  4. Run start.sh to start the Docker container

  5. Get the release version of the install script and run it with the created release branch `:

    $ curl -fsSLO https://tiny.cc/rc-rpi
$ bash rc-rpi release/1.X.Y

  6. Setup the VM

    $ bash -i setup-vm.sh release/1.X.Y

  7. Test download a ROM, scrape it, and validate that it shows up in Azure:

    $ bash -i retro-cloud-setup/dev/download-freeware-roms.sh
$ bash -i run-scraper.sh
$ bash -i retro-cloud-setup/dev/run-tests.sh

  8. Run the ROM

  9. If everything works so far (including greed CI build), continue creating the release

  10. Checkout master

    1. git merge release/1.X.Y --no-ff
    2. git push

  11. Checkout develop

    1. git merge master --no-ff
    2. git push

  12. Create a new release on Github and tag the latest master commit with v1.X.Y