Box to Azure Blob Storage Pipeline
Runtime Requirements
Python 3.6+
PIP package manager
virtualenv
Set up and Run
- From the project root folder, create a Python 3.6+ virtual environment
$ virtualenv --python=python3 env
- Activate the virtual environment
$ source env/bin/activate
- Install the project dependencies
$ pip install -r requirements.txt
- Copy the configuration from from
data/configuration/no_key_example.yml
to
data/configuration/dev.yml
- Add the Box and Azure platform related attributes to the configuration file
- Run the command line interface menu to see the available commands
$ python src/cli/main.py
- Run the seed-datetime-folders command to build a datetime chunked Box environment
$ python src/cli/main.py seed-folders
- Run the box-to-azure command to move files from an upload to chunked folder, then upload them to Azure Blob Storage
$ python src/cli/main.py box-to-azure
- Build the development Docker environment
$ make dev
- Build the production Docker environment
$ make prod