Beacons Migration Pipeline

Please see the Beacons Miro board for more information on the migration.

MAC setup

Ensure python 3.5+ is installed

Dependencies for the project are managed through pipenv. To install pipenv run:

python3 -m pip install pipenv

Connecting to the Oracle DB

The Oracle DB connection string is set in the legacy database helper file. To standup a local Oracle DB instance see the docs in the Beacons Oracle GitHub repo.

Installing Dependencies

To install the dependencies for the project run:

pipenv install --dev

The Oracle instantclient is required to connect to the Oracle instance.

To download the instantclient go to:

Download macOS instantclient
Unzip to directory ./instantclient_19_8 in this repository's directory
Run pipenv run python run_cleansing_rules.py

NOTE: When you run this for the first time macOS cannot verify the client files. You will have to open up: System Preferences > Security & Privacy and allow access to the files that cx_Oracle requires

Running the migration

To run the migration locally copy the .env configuration from the 1Password for the environment you want to run it against.

Without Docker ❌ 🐳

Standup a local Oracle DB instance (information above)

Ensure oracle_uri in your config (config.ini) is set to localhost

Run:

pipenv run python migration.py

With Docker ✅ 🐳 (🐌)

This method requires less set up, but currently it runs very slowly and only gets around 9000 records migrated before it stops.

Ensure you have authenticated with the AWS ECR registry. See the 1Password vault for more information.

Run:

./run-migration.sh

Running Unit Tests

PyTest is used for unit testing. To run all the unit tests run:

pipenv run pytest

IDE setup

If using PyCharm for your IDE setup, see the docs for configuring pipenv.

mcagov/beacons-migration