/bolt-api

Primary LanguageCSSMIT LicenseMIT

AcaiBolt API - 1.0.0-beta

This repo holds the microservice gluing together the bolt-wrapper, hasura, database and keycloak services.

Note: Use git clone --recurse-submodules <reponame> to pull dependencies while cloning, or git submodule update --init --recursive from within project dir to fetch dependencies after cloning.

Development

Everything needed to run bolt-api locally is contained in docker-compose file, it should be enough to just execute:

docker-compose up

This starts hasura on http://localhost:8080 and bolt-api on http://localhost:5000 and basic monitoring.

To start a minimal set of services (db+hasura) use:

docker-compose -f docker-compose-debug.yaml up

and launch bolt-api beforehand by yourself, this may be necessary to debug hasura-bolt communication.

To connect to hasura and have your changes recorded:

  • cd subsystems/hasura
  • hasura-cli console

That opens a proxy to hasura listening on http://localhost:9695 with bolt-api as it's remote-schema.

To call bolt-api endpoints either open builtin graphiql console at http://localhost:5000/graphql or open hasura's console at http://localhost:9695/api-explorer

Documentation is contained and self-served in graphql schema, use one of the graphiql interfaces (listed above) to peruse.

Graphql queries and mutations

Hasura

A large set of graphql methods is made available by hasura for CRUD operation without employing bolt-api where possible, these have their permissions set appropriately. For these permissions to work correctly, the user request must identify to hasura using jwt token prepared by keyclock per hasura documentation but with two required fields:

  • x-hasura-user-id - contains unique user UUID generated by keycloak/authenticator,
  • x-hasura-default-role - one of admin, manager or reader.

The UUID should be matched in user_projects table to grant access to specific projects.

Bolt-API

Where hasura methods are insufficient, bolt-api provides a superset of methods:

Queries:
  • testrun_repository_key - returns id_rsa.pub key used by bolt-api
Mutations:
  • testrun_project_create - high-level interface to create and validate a project
  • testrun_project_create_validate - as above, validation only
  • testrun_repository_create
  • testrun_repository_create_validate
  • testrun_repository_update
  • testrun_repository_update_validate
  • testrun_configuration_create
  • testrun_configuration_create_validate
  • testrun_configuration_update
  • testrun_configuration_update_validate
  • testrun_start - given a configuration id, start actual tests (pass result to testrun_status)

For current list check graphiql interface.

Authentication in development

Hasura setup in docker-compose uses a token literal of devaccess, use that to authenticate per Hasura docs.

Project organization

  • /apps/bolt_api/ - bolt-api flask server, forwarded as remote hasura chema, use flask run to start in development mode or ./wsgi.py
    • /apps/bolt_api/app/appgraph - contains graphql mutations and queries
    • /apps/bolt_api/app/webhooks - contains public (!) endpoints for hasura to call on configured events, list with flask routes
  • /apps/bolt_metrics_api/ - metrics exporter for offloading and separating heavy tasks
  • /charts/ - helm deployment charts
  • /cmds/ - helper flask commands list with flask --help
  • /instance/ - flask configuration, override with CONFIG_FILE_PATH and SECRETS_FILE_PATH env variables
  • /services/ - python module containing core functionality
  • /subsystems/ - holds hasura migrations, schema definitions, and hasura-cli configuration
  • /requirements.txt - installs development requirements

Production Setup & Deployment

Dockerfile in project root is a reference point for any deployment, with bolt-api configuration controlled by CONFIG_FILE_PATH and SECRETS_FILE_PATH env variables.

Deployment requirements:

  • python >= 3.6 venv
  • hasura
  • pip install -r requirements.txt
  • project configuration variables:
    • by default are read from /instance/localhost-config.py and /instance/secrets.py
    • conf files locations are configurable through CONFIG_FILE_PATH and SECRETS_FILE_PATH environment variables
    • missing but required config variables are checked at startup
    • external services necessary for operation are loaded eagerly where possible, with Hasura being notable exception (Hasura starts after bolt_api)
    • see /services/const.py:REQUIRED_BOLT_API_CONFIG_VARS for current requirements descriptions
    • access to Google services (buckets, etc) is facilitated by setting a GOOGLE_APPLICATION_CREDENTIALS environment variable to the path of a google service account json file

Manual Startup Sequence

Preparation

Hasura:
  • set HASURA_GRAPHQL_ACCESS_KEY equal to api's
  • set BOLT_API_GRAPHQL to bolt-api graphql address, eg. http://api:5000/graphql
  • set BOLT_API_CONFIGURATION_PARAM_CHANGE and BOLT_API_EXECUTION_STATE_CHANGE to full paths of webhooks, eg. http://api:5000/webhooks/execution/update
DB:

Once hasura and db are up, go into /subsystems/hasura/ subfolder, adjust endpoint in hasura/config.yaml and execute hasura CLI tool:

/bin/hasura migrate apply
Services:

Start db first, then api, then hasura, finally any monitoring.

Order is important.

Remote schema:

Hasura migrations (see DB) set up bolt-api as remote schema thus offloading access and authorization to hasura. Bolt-api queries and mutations are to be distinguished by the testrun_ prefix.

File Uploads

Overview:

Two GCS buckets are used:

  • const.BUCKET_PRIVATE_STORAGE - to temporarily keep uploaded files before processing
  • const.BUCKET_PUBLIC_UPLOADS - to eventually store processed uploaded files in a publicly accessible location.

Files are uploaded directly to temporary location in a object-lifetime-limited GCS buckets, through time-limited signed urls which can be generated by a call to testrun_upload mutation. It accepts file mimetype, file length, and base64 encoded file content MD5 hash.

In response user is returned a pair of signed urls, both pointing to a unique, non-existing resource located in a temporary and private Google Cloud Storage bucket. User responsibility is to subsequently PUT the file to bucket using upload_url and pass download_url to whichever mutation *_url argument should associate the upload with it's database-stored object.

Mutations which have *_url argument(s) (at the time of this writing only testrun_project_create/_update argument image_url):

  • accept a signed download url as value (generated above),
  • expect there to be a file in it's location (result of PUT to the upload_url),
  • process it and upload to target bucket if there is one, or
  • or fail quietly if it's missing, or
  • or, if the argument points anywhere else than the uploads bucket then the value will be stored in db as-is.

Processed images have a set of thumbnails, see /services/uploads/thumbnails.py for listing and description.

After successful processing, object's (eg. a project) image url will point to a set of resources in the public bucket:

  • http://storegadomain.com/<const.BUCKET_PUBLIC_UPLOADS>/RESOURCE_NAME - image in default thumbnail size (thumbnails.DEFAULT_SIZE)
  • http://storegadomain.com/<const.BUCKET_PUBLIC_UPLOADS>/RESOURCE_NAME/original - original image in it's full size, if file size does not exceed thumbnails.MAX_ORIGINAL_BYTES
  • http://storegadomain.com/<const.BUCKET_PUBLIC_UPLOADS>/RESOURCE_NAME/800x300 - image thumbnail that's 800px wide and 300px tall
  • http://storegadomain.com/<const.BUCKET_PUBLIC_UPLOADS>/RESOURCE_NAME/<WWW>x<HHH> - and so forth for each of thumbnail.SIZES

Upload from commandline:

To test uploads e2e:

flask upload_file --path /tmp/file.jpg

Then check for errors and if scaled versions are in fact available in configured buckets.

Upload steps example:

To perform an upload manually:

  • generate Content-MD5 standard header for file being uploaded:
$ cat file_to_upload.jpg | openssl dgst -md5 -binary  | openssl enc -base64

xoK7oR4ezmKgX243mDbWuw==
  • request an upload/download url from api:
$response = mutation{
  testrun_upload (
    content_length:123
    content_type:"image/jpeg"
    content_md5:"xoK7oR4ezmKgX243mDbWuw=="
  ) {
    returning {
      upload_url
      download_url
    }
  }
}
  • upload the file, eg.:
curl \
    -X PUT \
    -H "Content-Type: image/jpeg" \
    -H "Content-MD5: xoK7oR4ezmKgX243mDbWuw==" \
    -T - {$response.returning.upload_url} < file_to_upload.jpg
  • after successful upload pass $response.returning.upload_url to object mutation:
mutation {
  testrun_project_update(
    id:"d85d29e5-8204-46a7-8218-40bdcf68c978"
    image_url:"{$response.returning.download_url}"
  ) { affected_rows }
}

Users

Authentication is handled by Keycloak and passed onto Hasura through a JWT with special claims structure.

Below is a Script Mapper for generating hasura claims. It relies on calling client having a set of roles matching the ones used by Hasura and bolt-api, see sevice/const.ROLE_CHOICE.

/**
 * Available variables: 
 * user - the current user
 * realm - the current realm
 * token - the current token
 * userSession - the current userSession
 * keycloakSession - the current userSession
 */
var ArrayList = Java.type("java.util.ArrayList");
var forEach = Array.prototype.forEach;
var roles = new ArrayList();
var realmClients = realm.getClients(); 
var currentClient = keycloakSession.getContext().getClient();
var user_id = user.getId();
var default_role = "";
// TODO: bolt multitenancy support needs some support in keycloak too
var tenant_id = user.getFirstAttribute("tenant_id");

function mapper (roleModel) {
    if (!roles.contains(roleModel.getName())) {
        roles.add(roleModel.getName());
        if (roleModel.isComposite()) {
            forEach.call(roleModel.getComposites().toArray(), mapper);
        }
    }
}

forEach.call(user.getClientRoleMappings(currentClient).toArray(), mapper);

if (roles.contains("reader")) {
    default_role = "reader";
}
if (roles.contains("tester")) {
    default_role = "tester";
}
if (roles.contains("manager")) {
    default_role = "manager";
}
if (roles.contains("tenantadmin")) {
    default_role = "tenantadmin";
}
if (roles.contains("admin")) {
    default_role = "admin";
}

var claims = {
    "x-hasura-default-role": default_role,
    "x-hasura-tenant-id": tenant_id,
    "x-hasura-allowed-roles": roles,
    "x-hasura-user-id": user_id,
  };
  
exports = claims;

Debugging

You can debug API itself using latest version of PyCharm. First, a valid debug server needs to be running before you enable debugging in the API itself.

  1. Create new Python Debug Server configuration in PyCharm
  2. Set port to any open free port (use port 6060 if you don't want to specify custom port in step 7.)
  3. Set host to localhost
  4. Set path mapping so that project root is matched with literal docker root. For example, if your cloned repo root is in ~/Acai/bolt/bolt-api, then set path mapping to ~/Acai/bolt/bolt-api=/
  5. Save the configuration under the name of your choosing and run it in debug mode
  6. Open instance/localhost-config.py and add a new field named DEBUG_SERVER and set it to the literal IP address of native system that is parent to your running docker instance. For example DEBUG_SERVER = '192.168.241.130'
  7. (Optional) Add a new field named DEBUG_PORT and set the value to any port of your choosing as integer. For example DEBUG_PORT = 3434

DO NOT COMMIT THIS FILE 8. Restart bolt-api container or make any change to API script files while setting desired breakpoints, as autoreloading is enabled by default.

If running bolt-api instance is refusing to connect to your debug server, first check if you are running latest PyCharm version. If not, upgrade it and try again. Otherwise please open Python Debug Server configuration, copy suggested pydevd version and update it in core API requirements file, then add it to your commit.