destiny-manifest-manager
Database and Javascript scripts used to load the Destiny SQLite manifest file to a Postgres database.
Database setup
Documentation on setting up Postgres can be found here.
The manifest data is loaded in the manifest schema.
All database scripts and sample queries are located in the sql folder. The script database-install.sql shows the order of processing if loaded with psql.
The doc folder contains the basic data model for staging and processing the manifest. The data model was created using Navicat Data Modeler which can be found here. A PNG version of the data model has also been uploaded to the repository and is shown below.
Bungie.net
The API calls to bungie.net require a developer API key. The key can be acquired by visiting https://www.bungie.net/en/Application and documentation of the APIs can be found here.
Processing
Processing is handled by three AWS Lambda functions written in node.js. The functions are checkManifest, downloadManifest, and loadManifest.
The doc folder also contains the process flow for the manifest loader. The process flow was created in the draw.io desktop Chrome application. The application can be found here. A PNG version of the process flow has also been uploaded to the repository and is shown below.
Note: The S3 bucket utilized for storing the uncompressed manifest file has a lifecycle policy that automatically deletes the manifest file after seven days.
Orchestration
Orchestration is handled within Amazon Web Services utilizing AWS Step Functions within a state machine. The state machine has two possible outcomes. The first is to download and load a new manifest file.
The second is to simply cease processing if no new version of the manifest is found.
The state machine setup can be found here
Lastly, the state machine is executed every twenty minutes by an AWS CloudWatch event.