sdx-collect is a component of the Office for National Statistics (ONS) Survey Data Exchange (SDX) project which listens to a queue for survey data from the eQ Survey Runner.
On receipt it hands off to sdx-decrypt to decrypt the survey data, validates the result with sdx-validate and then stores the json by calling sdx-store. Once complete the survey is receipted via the appropriate receipting service.
This application presently installs required packages from requirements files:
requirements.txt: packages for the application, with hashes for all packages: see https://pypi.org/project/hashin/test-requirements.txt: packages for testing and linting
It's also best to use pyenv and pyenv-virtualenv, to build in a virtual environment with the currently recommended version of Python. To install these, see:
- https://github.com/pyenv/pyenv
- https://github.com/pyenv/pyenv-virtualenv
- (Note that the homebrew version of
pyenvis easiest to install, but can lag behind the latest release of Python.)
Once your virtual environment is set, install the requirements:
$ make buildTo test, first run make build as above, then run:
$ make testIt's also possible to install within a container using docker. From the sdx-collect directory:
$ docker build -t sdx-collect .The following envioronment variables can be set:
| Environment variable | Default | Description |
|---|---|---|
| SDX_DECRYPT_URL | http://sdx-decrypt:5000/decrypt |
URL of the sdx-decrypt service |
| SDX_VALIDATE_URL | http://sdx-validate:5000/validate |
URL of the sdx-validate service |
| SDX_RESPONSES_URL | http://sdx-store:5000/responses |
URL of the sdx-store service |
| SDX_COLLECT_SECRET | none | Key for decrypting messages from queue, must be the same as used for sdx-receipt |
| DAP_SOURCE_NAME | sdx_development |
Name of the environment the DAP message was created in |
| RABBIT_SURVEY_QUEUE | survey |
Name of incoming queue |
| RABBIT_QUARANTINE_QUEUE | survey_quarantine |
Name of queue to quarantine bad decrypt messages to |
| RABBIT_EXCHANGE | message |
Exchange for incoming queue |
| RABBIT_RRM_RECEIPT_QUEUE | rrm_receipt |
Name of rrm receipt service queue |
Copyright (c) 2016 Crown Copyright (Office for National Statistics)
Released under MIT license, see LICENSE for details.