This repo holds a reusable service for handling long-running HTTP Cloud Tasks that are meant to be sent to other GCP services such as GCF or Cloud Run which require an active HTTP connection.
The service is an App Engine service that works in pair with a Cloud Tasks queue.
This blend is unique for GCP while tasks that are targeting App Engine standard services with basic scaling can run for up to 24 hours. We're gonna be using this feature to perform long-running connections to other deployed services that require a live connection (e.g. Cloud Functions or Cloud Run).
Here's how such long-running requests may look like:
sequenceDiagram
actor user
participant cloudTasks as Cloud Tasks
participant tasksRunner as Tasks Runner (App Engine)
participant services as Services
user->>cloudTasks: schedules an HTTP call task
cloudTasks->>tasksRunner: sends AppEngine HTTP task
tasksRunner->>services: sends task payload via authenticated HTTP call
services->>tasksRunner: sends back a response
tasksRunner->>cloudTasks: sends back response details
It's obvious that in this schema there is no direct way to notify the caller
about the
HTTP call status so this part is on the service/caller to identify the result of a call.
E.g. one may supply some callback details in the request payload for the services
to send a
callback before returning HTTP response to the tasksRunner
.
The full setup of the required infrastructure is performed with a helper script
setup-env.sh
. The script enables required services, creates App Engine
application, service account, queue and secrets.
After the setup is done one may deploy the service using deploy.sh
script.
Just a couple of things one may want to improve into this service.
-
Add retry policy with some jitter.
Right now the service will just fail and/or rely on the Cloud Tasks retry policy which may not be always a desired behavior. Ideally one should be able to provide customized retry policy as part of the request payload and disable Cloud Tasks retries while they are not that universal.
-
Add some well-structured logging.
No logging support is implemented in the service which may make debugging and diagnostics somewhat hard. Ideally one should implement structured logging e.g. using Cloud Logging capabilities.
-
Add some tracing support.
Tracing async executions across multiple services and environments is not the easiest task so it's a good idea to have some tracing support available in the service.
-
Have support of callbacks as part of this service.
It is expected that one will want to get a result of the async call in one or another way. The service should provide some capabilities out of the box for doing some simple callbacks or have an endpoint that may allow checking up on the status of the execution.