topcoder-platform/legacy-challenge-migration-script

Migrates Challenges, Resources, and Resource Roles from Informix to DynamoDB

Legacy Challenge Migration CLI Tool

Migration script used to migrate Challenges, Resources, and Resource Roles from Informix to DynamoDB. It runs on a scheduled basis and also on-demand by exposing an API allowing admins to manually trigger the migration.

Development deployment status

Production deployment status

Intended use

• Data migration script to move v4/Informix to v5 Dynamo/ES

Related repos

• Challenge API
• Resources API

Prerequisites

• NodeJS
• Elasticsearch(v6.3.1)
• DynamoDB
• Informix
• Docker(CE 17+)
• Docker Compose

Configuration

See config/default.js. Most of them is self explain there.

• PORT: API server port; default to 3001
• API_VERSION: API version; default to v5
• SCHEDULE_INTERVAL: the interval of schedule; default to 5(minutes)
• CHALLENGE_TYPE_API_URL Challenge v4 api url from which challenge types data are fetched.
• CHALLENGE_TIMELINE_API_URL Challenge v5 api url from which challenge timelines are fetched.
• CREATED_DATE_BEGIN A filter; if set, only records in informix created after the date are migrated.
• BATCH_SIZE Maximum legacy will be load at 1 query
• ERROR_LOG_FILENAME Filename for data that error to migrate.
• RESOURCE_ROLE List of resource role to be included in migration

Local Deployment

Foreman Setup

To install foreman follow this link To know how to use foreman follow this link

If you run something with nf run npm run <something> instead of npm run <something>, it'll load in a .env file from the root of the project

Deployment

To simplify deployment, we're using docker. To build the images or run the container:

cd <legacy-challenge-migration-cli>/docker
docker-compose up

This will automatically build the image if have not done this before. After container has been run, go to container shell and install dependencies:

docker exec -ti legacy-challenge-migration-cli bash
npm i

Command for API

• Inside the docker container, start the express server: npm start

Migrations

For local migrations

There are two parts need to be updated for local migrations,

• ./src/models/challenge.js throughput: 'ON_DEMAND', should be updated to throughput:{ read: 4, write: 2 },
• ./config/default.js Two aws config should be uncommented

and env variable IS_LOCAL_DB should be set to true before you continue to the next steps.

Run a migration

• To run a migration, the command should be: MIGRATION=<name-of-migration-file> nf run npm run migrate Example: MIGRATION=001-migrate-taskData nf run npm run migrate

Production deployment

• TBD

Running tests

• TBD

Running tests in CI

• TBD