lookit/lookit-orchestrator

Issue with kustomize.yaml while deploying lookit

Closed this issue · 10 comments

rhodricusack commented

I've been trying to run through the deployment pipeline, and when I try either the manual or the automatic google cloud-build deployment, I get an error of

Step #7 - "deploy-to-cluster": Error: unable to find one of 'kustomization.yaml', 'kustomization.yml' or 'Kustomization' in directory '/kubernetes/lookit/environments/staging'

Its coming from line 57 in https://github.com/lookit/lookit-orchestrator/blob/master/deploy.sh I thnk

Do you know why this might be? I see there is a kustomization.yaml in kubernetes/lookit/base but there is lots of stuff in that directory too.

Thanks for the elegant system-

Datamance commented

Hi @rhodricusack - can you describe what you're trying to do? Are you trying to deploy lookit to your own cloud? You really shouldn't need this repo for anything; it's bespoke CI tooling for lookit-api.

If you're trying to develop Lookit locally, you should look into checking out lookit-api. Otherwise, I would chat with @kimberscott a bit so that we can understand what you're trying to do find out best how to help you get there.

kimberscott commented

@Datamance They are indeed actually trying to see how hard it would be to deploy their own instance of Lookit, to use for a large collaborative project where most labs would have restrictions on storing any data outside the EU. Ideally long-term we'd have a more unified platform, but this seemed like a reasonable approach as a start. I suspect, though, that they shouldn't need nearly as much CI pipeline machinery.

rhodricusack commented
kimberscott commented

Makes sense! For actual development, just working locally is by far the simpler and more efficient option, but I think there's value in being able to get feedback from less technical collaborators about how everything looks by deploying it on GCP. (And it might be good to have a sense of what would be involved in hosting your own instance if it turned out to be necessary.) I'll leave the setup questions to @Datamance though...

Datamance commented

@rhodricusack a good place to start is the readme, But to fill in a bit more:

General Architecture

High-level overview

I describe Lookit ecosystem as a sort of "restaurant" architecture, if the restaurant had a prix fixe menu. It's a cheeky and imperfect analogy, but it gets the general idea across. In this analogy there are a couple of main players you need to know about:

  1. experiments are meals
  • packaged/built web archives loaded into Google Cloud Storage (GCS) and proxied/served by the web application (see point 4 below)
  • Right now, the only experiment type we support is ember-lookit-frameplayer; theoretically we could create adapters for lookit for different experiment front-ends.
  1. researchers are head chefs
  • they design experiments and then build their dependencies with a process that uploads them to GCS.
  1. participants are customers
  • they use the wait staff to order meals that are on the menu.
  1. The web application is your wait staff
  • They show off the menu to customers
  • They also take orders from the head chef to add, remove, or change items on the menu, and relay those to the kitchen (see point 5 below)
  1. The builder is your kitchen
  • receives orders from the head chef by way of the wait staff.
  • actually cooks "meals" (builds experiments)
  • RabbitMQ kind of acts like a stack of order slips, in this system.
  1. The worker is kind of like a general manager:
  • sends out mail (email) to previous customers
  • occasionally, you'll see him sweeping the kitchen (cleaning up junk left over from docker builds)

The "Frameplayer" and video data

Ember-lookit-frameplayer is an ember application that can be "built" into a web archive (hereafter WAR) with bundled/minimized javascript and css. It's the default experiment type, and currently the only one available.

To capture video data, we use Pipe. The JS client is basically a plugin that is parameterized during the WAR build process; when properly configured it will stream video data to Pipe's servers. This data is then uploaded to Amazon S3, which, upon completion, triggers a webhook that fires a POST payload to a special handler in our API that finds the video in S3 and renames it to something more searchable.

A Note about Configuration

So far, we see quite a number of "in-network" players (webapp, builder, worker, etc.) and "out-of-network" services (Pipe, S3, Google Cloud Storage, Google Cloud SQL). While Kustomize and Kubernetes work tightly together to connect "in-network" players, "out-of-network" services all need login credentials. You will see how these credentials are safely and securely built into deployed k8s pods in the "Setup" section.

Cluster layout

Lookit is organized as a collection of Kubernetes services, backed by deployments and statefulsets:

❯ kubectl get services
NAME                               TYPE           CLUSTER-IP   EXTERNAL-IP              PORT(S)                                          AGE
kubernetes                         ClusterIP      10.x.x.x     <none>                   443/TCP                                          89d
staging-gcloud-sqlproxy            ClusterIP      10.x.x.x     <none>                   5432/TCP                                         89d
staging-google-storage             ExternalName   <none>       storage.googleapis.com   <none>                                           89d
staging-lookit-rabbitmq            ClusterIP      10.x.x.x     <none>                   4369/TCP,5672/TCP,25672/TCP,15672/TCP,9419/TCP   89d
staging-lookit-rabbitmq-headless   ClusterIP      None         <none>                   4369/TCP,5672/TCP,25672/TCP,15672/TCP            89d
staging-web                        NodePort       10.x.x.x     <none>                   8080:32403/TCP                                   89d

❯ kubectl get deployment
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
staging-beat              1/1     1            1           89d
staging-gcloud-sqlproxy   1/1     1            1           89d
staging-web               1/1     1            1           89d
staging-worker            1/1     1            1           89d

❯ kubectl get statefulset
NAME                      READY   AGE
staging-builder           1/1     89d
staging-lookit-rabbitmq   1/1     89d
  • The backing monorepo for the -web (web application), -builder (experiment builder), -worker (celery tasks), and -beat (celery crons) resources is lookit-api, which is a django application.
  • The -gcloud-sqlproxy resources define as a single point of egress out to a Google Cloud SQL instance, designated by the configuration file for a given environment (production example)
  • The -lookit-rabbitmq resources define a rabbitmq message queue that serves as a conduit between the web application and the task runner and builder.
  • -google-storage is basically just an external service that we set up to allow nginx ingress to reroute requests for static assets to GCS.

Setup

Google Cloud Platform (GCP)

We rely almost exclusively on GCP components to orchestrate the app. A Cloud Builder Github integration trigger is tripped on deployments to either the "develop" or "master" branches of lookit-api, executing the "CI" piece of the pipeline (testing and containerization). You can see in the deploy-to-cluster step that the "CD" (deployment) piece is executed near the very end. It leverages the contents using the contents of this repo, which are similarly containerized (using the GitHub integration for build triggers) and loaded into GCR for use as a custom builder.

I did not have the time to completely generalize/parameterize this CI pipeline, so to run your own lookit CI pipeline, you'll basically want to set up your own brand new environment apart from the one that is used by the MIT instance of Lookit. To accomplish this, you'll need to set up your own Google Cloud Platform project. You'll need a few things turned on:

  • Kubernetes Engine
  • Cloud Builder
  • Container Registry
  • Key Management Service

Once those services are turned on, you'll want to turn your focus to the GKE setup that is tuned by this repo.

Kubernetes setup (lookit-orchestrator configuration)

I recommend following these steps:

  1. Fork this repo, as well as lookit-api
  2. Change the configs to match your new environment (in fact, you will probably need to make changes to all of the files in that environment directory to suit your particular environment)
  3. Create a secrets file for your new environment (ignored such that it is never checked in). You'll author this in basically the same way as the config env file; to see which secrets you'll need you can take a look at the secretKeyRef'd env vars listed in the env var patch.
  4. Run one of the make encrypt hooks (here) to encrypt your plaintext secrets into .enc versions that can be checked into source control (thusly facilitating GitOps deployment - you can see in the deploy script where the secrets are actually decrypted using GKMS.

Now, given that I didn't have time to generalize/parameterize this pipeline to target arbitrary clusters, you'll also need to edit the actual deployment line of the script to target whatever zone/region you're deploying to in Europe. There are probably other things that I'm missing at the moment (Sentry setup, for instance, is baked into the app and CI/CD pipeline - you'll need to figure out if you want alerting and modify accordingly), but this should give you a decent start. It'll be a good idea for us to work together closely to both get you up and running, and also provide critical feedback in making this pipeline more adaptable.

Datamance commented

@rhodricusack, is it ok if I close this out?

rhodricusack commented
jtrouth commented

@rhodricusack If you're still working on this, could you provide the full output of the deploy-to-cluster Cloud Build step? It's strange that the kustomization YAML from the staging directory wouldn't be found, but is most likely an issue with either the lookit-orchestrator container image or with the path of the manifests directory being passed to customize in the deploy.sh script.

mekline commented

Spring cleaning @rhodricusack , can we close this issue?

rhodricusack commented