Basic Hasura-based API as a service
rufuspollock opened this issue · 19 comments
Epic here #1
When developing data-api I want to be able to create and run integration tests of the app so that I can check how it's working together with postgres and hasura.
Acceptance
- There is a docker-compose environment with the necessary cervices
- There is ci running tests and publishing to a Docker repository
Tasks
Local developer docker-compose
- add services to docker-compose
- postres (latest official postgres image)
- hasura (custom image with adding sample data)
- write a basic smoke test checking the system parts are working
- Find the framework to use (Supertest remains the default for node http server tests ...)
CI
- Dockerfile (
FROM nodejs ...+ copy files) - Publish Docker image (used https://github.com/datopian/data-api/new/master?filename=.github%2Fworkflows%2Fdocker-publish.yml&workflow_template=docker-publish for how to)
- Add another docker-compose for ci
- add data-api image building from source (include tests)
- check code correctness in CI on every push:
- run prettier
- run tests
- create new version Docker tags automatically - CI on every push to master:
- update version (if tests ok)
- publish new version of the image
- publish :latest tag with the new changes
Inbox
- Have Hasura based read API
- Dockerize this service and auto publish on merge to master
- Work out where we publish to (is it dockerhub or is it github packages? (Ask on tech help?)
- Publish there with continuous deployment (on master)
- Testing
- Have basic test for this (using cypress (?))
- CI setup and passing
@rufuspollock I don't think we really need to store hasura settings anywhere or publish a custom hasura image, because what we will set up in hasura is
- mark which tables we want to track (hasura metadata)
- run any sql scripts preparing tables/views (hasura migrations)
That is very project specific, I don't think we can have anything Datopian wide here. We can just launch hasura service in a project (like we would launch postgres):
Example Hasura Dockerfile
FROM hasura/graphql-engine:latest.cli-migrations-v2
COPY migrations /hasura-migrations/
COPY metadata /hasura-metadata/Example docker-compose containing Postgres, Hasura, A third service
version: '3'
services:
wrapper:
image: my-app-wrapper
build:
context: wrapper/
dockerfile: Dockerfile
graphql-engine:
image: my-app-hasura
build:
context: hasura/
dockerfile: Dockerfile
ports:
- "8080:8080"
depends_on:
- db
restart: always
environment:
HASURA_GRAPHQL_DATABASE_URL: postgres://ckan_default:password@db:5432/datastore_default
## enable the console served by server
HASURA_GRAPHQL_ENABLE_CONSOLE: "true" # set to "false" to disable console
## enable debugging mode. It is recommended to disable this in production
HASURA_GRAPHQL_DEV_MODE: "true"
HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log
## uncomment next line to set an admin secret
# HASURA_GRAPHQL_ADMIN_SECRET: myadminsecretkey
db:
image: my-app-postgres
build:
context: postgres/
dockerfile: Dockerfile
ports:
- "5434:5432"
environment:
POSTGRES_DB: datastore_default
POSTGRES_USER: ckan_default
POSTGRES_PASSWORD: passwordWON'T FIX. See above comments.
@EvgeniiaVak what i would be doing here is listing the docker compose setup you have described in e.g. the README and actually setting that up for this project with CI - so we have a basic test for running hasura against a sample DB (i know this may duplicate some work elsewhere but doing it here should be trivial and is useful IMO).
Tests are passing in docker runner:
...
$ mocha -r dotenv/config --recursive
data-api
�[0mGET /non_existing_page �[33m404 �[0m4.439 ms - 10�[0m
✓ should return 404 for non existing page (40ms)
GraphQL endpoint
�[0mPOST /v1/graphql �[32m200 �[0m25.855 ms - -�[0m
✓ returns graphql schema (61ms)
2 passing (112ms)
...
Prettier check is running on new builds:
Step 9/10 : RUN ["yarn", "prettier", "--check", "."]
---> Running in 79016bfb81d4
yarn run v1.22.4
$ /usr/src/app/node_modules/.bin/prettier --check .
Checking formatting...
All matched files use Prettier code style!
@EvgeniiaVak great - can we tick acceptance then and close?
@rufuspollock let's not close it just yet, I really want to make this automatic version tagging work (right now the it's always data-api:latest), also github seems to run tests too, and they fail (I think additionally to dockerhub (which pass))
When trying to push a change to the github workflow file (.github/workflows/docker-publish.yml) from local machine there is an error:
evgeniyavakarina (master) data-api $ git push
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 4 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (5/5), 624 bytes | 624.00 KiB/s, done.
Total 5 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com/datopian/data-api.git
! [remote rejected] master -> master (refusing to allow an OAuth App to create or update workflow `.github/workflows/docker-publish.yml` without `workflow` scope)
error: failed to push some refs to 'https://github.com/datopian/data-api.git'
Using GitHub UI for that for now as a solution.
@zelima I think I need your help here.
I created a github action (as from this example https://github.com/datopian/data-api/new/master?filename=.github%2Fworkflows%2Fdocker-publish.yml&workflow_template=docker-publish) - .github/workflows/docker-publish.yml but tests always failed because I couldn't quite set up environment on github correctly, so I deleted it.
On dockerhub tests always run (even without ci on github) and they run successfully example of the last one - https://hub.docker.com/repository/registry-1.docker.io/datopian/data-api/builds/d4993ba9-d577-413c-aa17-bfcd9f3d4705 (it's without any ci set up in github already).
What I think was happening: both github and dockerhub have ci running (I guess in dockerhub it's the one you've set up during our call @zelima ).
What I think should be happening: only github ci running:
- tests
- updating app version and creating a corresponding docker image tag (if tests passed)
- publishin
:latestandvXtagged images on dockerhub
Maybe I'm wrong here and only dockerhub ci should be running. Please let me know if we have any examples at Datopian already set up correctly so I could create a similar setup.
@EvgeniiaVak I'm not sure about the details. But probably docker is not running any tests, unless we somehow tell it. Ther eporbably is a way the we did not setup. So my guess is it'sonly running the build part.
I'm can't remember any Datopian project that does that
@zelima no, it's running the tests: e.g. see this one https://hub.docker.com/repository/registry-1.docker.io/datopian/data-api/builds/d4993ba9-d577-413c-aa17-bfcd9f3d4705. it has all the logs from mocha tests:
$ mocha -r dotenv/config --recursive
data-api
�[0mGET /non_existing_page �[33m404 �[0m4.667 ms - 10�[0m
✓ should return 404 for non existing page (43ms)
GraphQL endpoint
�[0mPOST /v1/graphql �[32m200 �[0m28.146 ms - -�[0m
✓ returns graphql schema (64ms)
2 passing (123ms)
Done in 0.79s.
here is the documentation for it - https://docs.docker.com/docker-hub/builds/automated-testing/
And it would be perfect to live it like this (since ci runs as expected but on dockerhub, rather than github) but I don't see a way to create version tags for images there...
@rufuspollock I think we should close this issue and create another additional one.
What we have now - every time there is a push to master branch tests run on dockerhub and if pass a new docker image (tagged :latest) is built (I think it meets the criteria from the inbox).
I think we still need versioning but since it's not so straightforward let's create a separate issue. And another nice thing would be to have tests running on pull requests to master also.
@EvgeniiaVak I've just looked in docker build configurations and seems like you can configure it so that it builds from tags.
That's on docker side. From github, you can either simply push a new release/tag manually. Or maybe even write small script to run that with travis.
Also just remembered think ckan-cloud-helm is deploying to dockerhub after succefull tests https://github.com/datopian/ckan-cloud-helm/blob/master/.travis.yml
Thank you @zelima that's really helpful!
So if we go with builds from tags, we would still need to create those tags first (manually?)... 🤔
So if we go with builds from tags, we would still need to create those tags first (manually?)...
Yes. Usually, that's done manually. But think you can easily scrip it if you really want. Or maybe even github has some tools to do that for you
Ok. Tests on pull requests are also running (without building a new image)! 🎉
I created a test PR from branch test-ci. Pushed a commit that adds a file. Then pushed a commit that deletes the test file. On the page https://hub.docker.com/repository/docker/datopian/data-api/builds. We can see 2 builds from test-ci branch:
FIXED. Test setup is in docker-compose.test.yml file (as per Dockerhub docs https://docs.docker.com/docker-hub/builds/automated-testing/). CI jobs here https://hub.docker.com/repository/docker/datopian/data-api/builds.