How to deploy locally (not AWS)
Opened this issue · 4 comments
Apologies for the likely naive question. I have a local, test STAC-API compliant instance running on in a docker cluster (minio for the holding STAC compliant JSON files, Apache, etc.) and then have STAC-Browser serving these files out in the nice web instance.
I am interested in testing sat-api without pushing to AWS. So I have also set up an ES cluster and pushed the stac metadata into that. How do I go about deploying the sat-api locally?
My current attempts to use a simple docker-compose file with the API are failing and I am just guessing at the environment variables and the ports that I need to populate/expose.
version: '3'
services:
satapi:
image: 'satutils/sat-api:develop'
ports:
- '9380:80'
environment:
STAC_ID: THEMIS
ES_HOST: smalls:9300
networks:
- default
@jlaura not a naive question at all. A while back this was an express API (easier to run locally) but that's deprecated now that we've moved to API Gateway. Because we aren't using SAM or serverless, there aren't great options to run this locally out of the box. Likely the "easiest" (but not that easy) way to run the API locally is by following the deploy instructions here until the deploy step and then running kes compile
instead of kes deploy
. This will give you a cloudformation template which you can use with localstack. There's a chance @matthewhanson is working on updated deployment steps which simplify this a bit as well
Hi @jlaura , you might want to check out the new STAC-API: https://github.com/stac-utils/stac-api
It's a completely refactored version of sat-api, and is compliant with the latest version of STAC, 0.9.0.
I have not deployed it locally, but it uses serverless so should be easier to set up.
@matthewhanson Thanks for the link. I have been able to make some more progress getting this running locally.
I wanted to post some info in case it is of use for anyone else trying to set this up locally.
Minio(S3) / Apache are running to store / serve the STAC compliant metadata, COGs, and thumbnail jpegs using the following docker-compose. Note that I am also using traefik locally, but this can be removed / omitted.
Minio/Apache docker-compose.yml
version: '3'
services:
apache:
image: 'bitnami/apache:latest'
ports:
- '9080:8080'
- '9443:8443'
volumes:
- ../single_image/:/app
labels:
- "traefik.enable=true"
- "traefik.frontend.priority=600"
- "traefik.frontend.rule=Method:GET"
- "traefik.frontend.rule=PathPrefixStrip:/stac"
- "traefik.backend=apache"
- "traefik.docker.network=traefik-public"
networks:
- default
- traefik-public
minio:
image: 'minio/minio:latest'
ports:
- 9000:9000
volumes:
- ./minio_data/:/data
labels:
- "traefik.enable=true"
- "traefik.frontend.priority=600"
- "traefik.frontend.rule=Method:GET"
- "traefik.frontend.rule=PathPrefixStrip:/minio"
- "traefik.backend=minio"
- "traefik.docker.network=traefik-public"
networks:
- default
- traefik-public
command: server /data
environment:
MINIO_ACCESS_KEY: minio
MINIO_SECRET_KEY: minio123
networks:
traefik-public:
external : true
We then need an elasticsearch backend to get the metadata pushed into. I am running this in yet another docker-compose:
ElasticSearch docker-compose.yml
1,1 All
version: '3'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.8.7
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
ports:
- "9300:9200"
kibana:
image: docker.elastic.co/kibana/kibana:6.3.2
ports:
- "9601:5601"
I am currently working on getting the ingest and api lambdas exposed as well in this setup. Luckily, lambci offers containers that I think will work.
Lambda docker-compose.yml
version: '3'
services:
lambda_ingest:
image: lambci/lambda:nodejs12.x
environment:
DOCKER_LAMBDA_STAY_OPEN : 1
ES_HOST : http://smalls:9300/
ports:
- "9001:9001"
networks:
- default
volumes:
- ./lambdas/ingest/dist:/var/task:ro,delegated
- ./libs:/opt
command: index.handler
lambda_api:
image: lambci/lambda:nodejs12.x
environment:
DOCKER_LAMBDA_STAY_OPEN : 1
ES_HOST : http://smalls:9300/
SATAPI_URL: http://smalls:9002/
ports:
- "9002:9001"
networks:
- default
volumes:
- ./lambdas/api/dist:/var/task:ro,delegated
command: index.handler
Using curl, I can do an ingest without a problem: curl -d '{"url":"http://127.0.0.1/stac:9000/themis/test/collection.json"}' http://127.0.0.1:9001/2015-03-31/functions/myfunction/invocations
and can see in the ES logs that things are being written. It is also possible to see the return on the call to the api
function: curl http://127.0.0.1:9002/2015-03-31/functions/myfunction/invocations
.
Next step is to test sat-search. I am running into a few issues here that I wonder if others might have insight:
- The API_URL needs to be set to some other address than the default. I tried using the environment variable and using an os.environ setter before import of sat search, but no joy. I had to modify the config.py. The variables were being set properly in search.py, but somewhere deeper in (urllib and ssl) the API URL was defaulting back.
- The stac/search endpoint is 404ing. I don't know if this is caller error or setup on the lambci side where each container needs to be a different container. I think the former since the determine endpoint function in the stac-api should be handling dispatching. Are stac-search and the stac-api in sync with each other for how URLs are being built? I'm on the stac 0.9.0 compliant branch and the git repo dev for stac-search.
Let me know if this info is helpful or if you all prefer users to focus on doing an AWS deploy.
@drewbo @jlaura A bit late to the party but I am putting the finishing touches on a SAT-API stack running on a local openfaas /kubernetes cluster with a local ES/KB deployment: sat-api-openfaas. My notes aren't finished yet but it does appear to work per spec with the Sentinel-5P dataset hosted on AWS. Please let me know if you have any questions -- thanks!