This repo holds the microservice gluing together the bolt-wrapper, hasura, database and keycloak services.
Note: Use git clone --recurse-submodules <reponame>
to pull dependencies
while cloning, or git submodule update --init --recursive
from within project dir
to fetch dependencies after cloning.
Everything needed to run bolt-api locally is contained in
docker-compose
file, it should be enough to just execute:
docker-compose up
This starts hasura on http://localhost:8080 and bolt-api on http://localhost:5000 and basic monitoring.
To start a minimal set of services (db+hasura) use:
docker-compose -f docker-compose-debug.yaml up
and launch bolt-api beforehand by yourself, this may be necessary to debug hasura-bolt communication.
To connect to hasura and have your changes recorded:
cd subsystems/hasura
hasura-cli console
That opens a proxy to hasura listening on http://localhost:9695 with bolt-api as it's remote-schema.
To call bolt-api endpoints either open builtin graphiql console at http://localhost:5000/graphql or open hasura's console at http://localhost:9695/api-explorer
Documentation is contained and self-served in graphql schema, use one of the graphiql interfaces (listed above) to peruse.
A large set of graphql methods is made available by hasura for CRUD operation without employing bolt-api where possible, these have their permissions set appropriately. For these permissions to work correctly, the user request must identify to hasura using jwt token prepared by keyclock per hasura documentation but with two required fields:
x-hasura-user-id
- contains unique user UUID generated by keycloak/authenticator,x-hasura-default-role
- one ofadmin
,manager
orreader
.
The UUID should be matched in user_projects
table to grant access to specific projects.
Where hasura methods are insufficient, bolt-api provides a superset of methods:
testrun_repository_key
- returns id_rsa.pub key used by bolt-api
testrun_project_create
- high-level interface to create and validate a projecttestrun_project_create_validate
- as above, validation onlytestrun_repository_create
testrun_repository_create_validate
testrun_repository_update
testrun_repository_update_validate
testrun_configuration_create
testrun_configuration_create_validate
testrun_configuration_update
testrun_configuration_update_validate
testrun_start
- given a configuration id, start actual tests (pass result totestrun_status
)
For current list check graphiql interface.
Hasura setup in docker-compose uses a token literal of devaccess
, use that to authenticate per Hasura docs.
/apps/bolt_api/
- bolt-api flask server, forwarded as remote hasura chema, useflask run
to start in development mode or./wsgi.py
/apps/bolt_api/app/appgraph
- contains graphql mutations and queries/apps/bolt_api/app/webhooks
- contains public (!) endpoints for hasura to call on configured events, list withflask routes
/apps/bolt_metrics_api/
- metrics exporter for offloading and separating heavy tasks/charts/
- helm deployment charts/cmds/
- helper flask commands list withflask --help
/instance/
- flask configuration, override withCONFIG_FILE_PATH
andSECRETS_FILE_PATH
env variables/services/
- python module containing core functionality/subsystems/
- holds hasura migrations, schema definitions, and hasura-cli configuration/requirements.txt
- installs development requirements
Dockerfile in project root is a reference point for any deployment, with bolt-api configuration
controlled by CONFIG_FILE_PATH
and SECRETS_FILE_PATH
env variables.
Deployment requirements:
- python >= 3.6 venv
- hasura
pip install -r requirements.txt
- project configuration variables:
- by default are read from
/instance/localhost-config.py
and/instance/secrets.py
- conf files locations are configurable through
CONFIG_FILE_PATH
andSECRETS_FILE_PATH
environment variables - missing but required config variables are checked at startup
- external services necessary for operation are loaded eagerly where possible, with Hasura being notable exception (Hasura starts after bolt_api)
- see
/services/const.py:REQUIRED_BOLT_API_CONFIG_VARS
for current requirements descriptions - access to Google services (buckets, etc) is facilitated by setting a
GOOGLE_APPLICATION_CREDENTIALS
environment variable to the path of a google service account json file
- by default are read from
- set
HASURA_GRAPHQL_ACCESS_KEY
equal to api's - set
BOLT_API_GRAPHQL
to bolt-api graphql address, eg.http://api:5000/graphql
- set
BOLT_API_CONFIGURATION_PARAM_CHANGE
andBOLT_API_EXECUTION_STATE_CHANGE
to full paths of webhooks, eg.http://api:5000/webhooks/execution/update
Once hasura and db are up, go into /subsystems/hasura/
subfolder,
adjust endpoint
in hasura/config.yaml
and execute hasura CLI tool:
/bin/hasura migrate apply
Start db first, then api, then hasura, finally any monitoring.
Order is important.
Hasura migrations (see DB) set up bolt-api as remote schema thus offloading access and authorization
to hasura. Bolt-api queries and mutations are to be distinguished by the testrun_
prefix.
Two GCS buckets are used:
const.BUCKET_PRIVATE_STORAGE
- to temporarily keep uploaded files before processingconst.BUCKET_PUBLIC_UPLOADS
- to eventually store processed uploaded files in a publicly accessible location.
Files are uploaded directly to temporary location in a object-lifetime-limited GCS buckets, through time-limited
signed urls which can be generated by a call to testrun_upload
mutation. It accepts file mimetype, file length,
and base64 encoded file content MD5 hash.
In response user is returned a pair of signed urls, both pointing to a unique, non-existing resource located in a temporary and private Google Cloud Storage bucket. User responsibility is to subsequently PUT the file to bucket using upload_url and pass download_url to whichever mutation *_url argument should associate the upload with it's database-stored object.
Mutations which have *_url argument(s) (at the time of this writing only testrun_project_create/_update argument image_url):
- accept a signed download url as value (generated above),
- expect there to be a file in it's location (result of PUT to the upload_url),
- process it and upload to target bucket if there is one, or
- or fail quietly if it's missing, or
- or, if the argument points anywhere else than the uploads bucket then the value will be stored in db as-is.
Processed images have a set of thumbnails, see /services/uploads/thumbnails.py
for listing and description.
After successful processing, object's (eg. a project
) image url will point to a set of resources in the public bucket:
http://storegadomain.com/<const.BUCKET_PUBLIC_UPLOADS>/RESOURCE_NAME
- image in default thumbnail size (thumbnails.DEFAULT_SIZE
)http://storegadomain.com/<const.BUCKET_PUBLIC_UPLOADS>/RESOURCE_NAME/original
- original image in it's full size, if file size does not exceedthumbnails.MAX_ORIGINAL_BYTES
http://storegadomain.com/<const.BUCKET_PUBLIC_UPLOADS>/RESOURCE_NAME/800x300
- image thumbnail that's 800px wide and 300px tallhttp://storegadomain.com/<const.BUCKET_PUBLIC_UPLOADS>/RESOURCE_NAME/<WWW>x<HHH>
- and so forth for each ofthumbnail.SIZES
To test uploads e2e:
flask upload_file --path /tmp/file.jpg
Then check for errors and if scaled versions are in fact available in configured buckets.
To perform an upload manually:
- generate
Content-MD5
standard header for file being uploaded:
$ cat file_to_upload.jpg | openssl dgst -md5 -binary | openssl enc -base64
xoK7oR4ezmKgX243mDbWuw==
- request an upload/download url from api:
$response = mutation{
testrun_upload (
content_length:123
content_type:"image/jpeg"
content_md5:"xoK7oR4ezmKgX243mDbWuw=="
) {
returning {
upload_url
download_url
}
}
}
- upload the file, eg.:
curl \
-X PUT \
-H "Content-Type: image/jpeg" \
-H "Content-MD5: xoK7oR4ezmKgX243mDbWuw==" \
-T - {$response.returning.upload_url} < file_to_upload.jpg
- after successful upload pass
$response.returning.upload_url
to object mutation:
mutation {
testrun_project_update(
id:"d85d29e5-8204-46a7-8218-40bdcf68c978"
image_url:"{$response.returning.download_url}"
) { affected_rows }
}
Authentication is handled by Keycloak and passed onto Hasura through a JWT with special claims structure.
Below is a Script Mapper for generating hasura claims. It relies on calling client having a set of roles
matching the ones used by Hasura and bolt-api, see sevice/const.ROLE_CHOICE
.
/**
* Available variables:
* user - the current user
* realm - the current realm
* token - the current token
* userSession - the current userSession
* keycloakSession - the current userSession
*/
var ArrayList = Java.type("java.util.ArrayList");
var forEach = Array.prototype.forEach;
var roles = new ArrayList();
var realmClients = realm.getClients();
var currentClient = keycloakSession.getContext().getClient();
var user_id = user.getId();
var default_role = "";
// TODO: bolt multitenancy support needs some support in keycloak too
var tenant_id = user.getFirstAttribute("tenant_id");
function mapper (roleModel) {
if (!roles.contains(roleModel.getName())) {
roles.add(roleModel.getName());
if (roleModel.isComposite()) {
forEach.call(roleModel.getComposites().toArray(), mapper);
}
}
}
forEach.call(user.getClientRoleMappings(currentClient).toArray(), mapper);
if (roles.contains("reader")) {
default_role = "reader";
}
if (roles.contains("tester")) {
default_role = "tester";
}
if (roles.contains("manager")) {
default_role = "manager";
}
if (roles.contains("tenantadmin")) {
default_role = "tenantadmin";
}
if (roles.contains("admin")) {
default_role = "admin";
}
var claims = {
"x-hasura-default-role": default_role,
"x-hasura-tenant-id": tenant_id,
"x-hasura-allowed-roles": roles,
"x-hasura-user-id": user_id,
};
exports = claims;
You can debug API itself using latest version of PyCharm. First, a valid debug server needs to be running before you enable debugging in the API itself.
- Create new Python Debug Server configuration in PyCharm
- Set port to any open free port (use port
6060
if you don't want to specify custom port in step 7.) - Set host to localhost
- Set path mapping so that project root is matched with literal docker root.
For example, if your cloned repo root is in
~/Acai/bolt/bolt-api
, then set path mapping to~/Acai/bolt/bolt-api=/
- Save the configuration under the name of your choosing and run it in debug mode
- Open
instance/localhost-config.py
and add a new field namedDEBUG_SERVER
and set it to the literal IP address of native system that is parent to your running docker instance. For exampleDEBUG_SERVER = '192.168.241.130'
- (Optional) Add a new field named
DEBUG_PORT
and set the value to any port of your choosing as integer. For exampleDEBUG_PORT = 3434
DO NOT COMMIT THIS FILE
8. Restart bolt-api container or make any change to API script files while setting desired breakpoints,
as autoreloading is enabled by default.
If running bolt-api instance is refusing to connect to your debug server, first check if you are running latest PyCharm
version. If not, upgrade it and try again. Otherwise please open Python Debug Server configuration, copy suggested pydevd
version and update it in core API requirements file, then add it to your commit.