apiarian/migration-playground

PICKLED

This repository is currently pickled. It's as far along as it will be for a while. It was started when I thought that the migration process I was exploring needed to be more complicated than it really did. Since the migration process is actually fairly straightforward (due to the fact that there isn't a real multi-master situation), there doesn't seem to be a need for this toy implementation anymore. Maybe it can be revived some day. Or maybe it can serve as a repository of code snippets. Anyway, probably won't be many updates here for a long while.

Migration Playground

A place to run experiments on migrating from one API backend to another using Kafka topics to choreograph the multi-master dance. Except we don't really have a multi-master dance. Since there's no way to merge changes between systems (the update operations are actually compare-and-set), we can just proxy requests to the current official master system internally.

Local Kafka

Setup

1. Download the Confluent Open Source Platform.
2. Extract the tar or zip file in the ~/development/kafka directory
3. Make a symlink to the latest confluent directory at ~/development/kafka/latest

Running Kafka

1. Open three terminal windows at the migration-playground directory
2. Run make zookeeper in the first and wait for zookeeper load
3. Run make kafka in the second and wait for kafka to load
4. Run make registry in the third and wait for the schema registry to load
5. When you're ready to shut things down, send an Interrupt (^C) to each of the terminal windows in reverse order

Original API

The Original API, found in the original-api directory, is a traditional permanent-storage-backed API. It handles Thing entities with list/get/create/update actions. Entities are stored in a persistent storage medium of some kind. The API also publishes the latest state of each entity to a kafka topic.

Running the Original API

The API can be launched by executing go run original-api/*.go.

See go run original-api/*.go -help for command-line arguments.

NOTE: Kafka needs to be available for the API to function.

Schema

Thing entities have the following schema:

{
	id: integer
	name: string
	foo: integer
	created_on: string(timestamp)
	updated_on: string(timestamp)
	version: integer
}

The name and foo fields are required when creating Things with the POST api/things/ endpoint or updated with the POST api/things/:id endpoint. The version field is also required when updating a Thing, and must match the current value of the version. The other fields are read-only and will be available at GET api/things/ and GET api/things/:id for getting a single Thing and listing all the Things, respectively.

Error responses have the following schema:

{
	error-message: string
}

Error responses have non-200 HTTP status codes.

application/json in, application/json out.

Shiny API

The Shiny API, found in the shiny-api directory, is a new stream-backed API. It handles Thing entities with list/get/create/update actions. It also handles command entities with get actions. Entities are kept in a kafka stream and cached locally as needed. The system is broken into two parts, the API itself shiny-api/api and the Updater tool shiny-api/updater. The API creates commands and watches for responses on a kafka topic, while the updater processes the business logic and deals with any data coordination that may be required with the Original API.

Running the Shiny API

The API command can be launched by executing go run shiny-api/api/*.go. The

See the -help output for command-line arguments.

NOTE: Kafka needs to be available for the API to function.

Schema

Thing entities have the following schema:

{
	id: string (numeric)
	name: string
	foo: float
	created_on: string(timestamp)
	updated_on: string(timestamp)
	version: string (numeric)
}

The name and foo fields are required when creating Things with the POST api/things/ endpoint or updated with the POST api/things/:id endpoint. The version field is also required when updating a Thing, and must match the current value of the version. The other fields are read-only and will be available at GET api/things/ and GET api/things/:id for getting a single Thing and listing all the Things, respectively.

NOTE: schema is similar and mappable (with slight loss in the foo field) to the Original API. Even though the id and the version fields have been turned into "opaque strings", they will need to be numeric for the duration of the multi-master dance of the two APIs.

Error responses have the following schema:

{
	error-message: string
}

Error responses have non-200 HTTP status codes.

application/json in, application/json out.

Running Everything

Run the Original API, Shiny API, and Updater all coordinated with the right topic names using ./run-apis.sh.

NOTE: the script is rather aggressive.