ReVal

ReVal (Reusable Validation Library) is a Django App for validating data via an API and web interface.

In the web interface, ReVal can manage data submitted as file uploads to a central gathering point, including data validation, basic change tracking and duplicate file handling. Each file generally contains multiple data rows, and each user may submit multiple files.

Using the API, ReVal can validate data and generally do anything that is possible via the web interface.

Features

Flexible input format
Validation with:
- goodtables
- JSON Logic
- SQL
- JSON Schema
- a custom validation class
Row-by-row feedback on validation results
Manage and track status of data submissions
Re-submit previous submissions
Flexible ultimate destination for data
API for validation

Requirements

Python (3.5, 3.6)
Django (2.2+)
Goodtables
pyyaml
djangorestframework
psycopg2
json_logic_qubit
dj-database-url
requests

Quick Installation

If you would like to use ReVal in your Django project, install using pipenv in your project...

Replace <version> with the latest tag i.e. v0.7.0 or
Replace with master if you would like to work with the latest development version

pipenv install -e git+https://github.com/18F/ReVAL.git@<version>#egg=data-ingest

Add 'rest_framework', and 'data_ingest' to your INSTALLED_APPS setting.

INSTALLED_APPS = (
    ...
    'rest_framework',
    'data_ingest',
)

Please see default installation for more setup instructions.

Examples

Several examples are provided to demonstrate default and customized behavior of data_ingest. Follow the development instructions to close this repository and install the dependencies required for the examples.

API

To perform data validation with API, see API documentation.

Deployment on Cloud.gov

All of the examples provided will show you how to run them locally. If you are interested in using cloud.gov as your platform, here's a basic installation guide on cloud.gov deployment.

Contributing

See CONTRIBUTING for additional information.

Development

To start developing on Django Data Ingest, clone the repository:

git clone git@github.com:18f/ReVAL.git

Install development dependencies:

pipenv install --dev

If you run into any issues installing packages in the Pipfile, you can try to install the particular package individually again.

Test Suite

To execute the test suite, install the development dependencies and run:

python runtests.py

Public domain

This project is in the worldwide public domain. As stated in CONTRIBUTING:

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

18F/ReVAL