Dataset Container

This repository allows building containers containing a file based dataset.

Usage

Use makefile image or push targets and provider necessary arguments:

SOURCE_DATA_DIR - path to a directory with a dataset, assumed to be a name of a dataset
DATA_ARCHIVE - path to an archived dataset used to calculated md5sum
DESCRIPTION - short description of a dataset

Example:

make \
  SOURCE_DATA_DIR=clinical-trials-data-800k \
  DATA_ARCHIVE=clinical-trials-data-800k.tar.xz \
  DESCRIPTION="Clinical trials studies and data objects" \
  all

The images are pushed to data-container repository on hub.docker.com.

Attached Metadata

Each data container has matadatach attached to it in form of images labels and container environment variables. The list of metadata attached:

DATA_DIR - location of a dataset in a container
NUMBER_OF_DIRECTORIES - numer of directories inside of DATA_DIR (excluding DATA_DIR)
NUMBER_OF_FILES - number of files inside of DATA_DIR
DESCRIPTION - short description of a dataset

Docker Image Naming Convention

Each dataset is published with 2 image tags latest and identified with it's md5sum, according to templates:

onedata/data-container: <dataset name> - latest
onedata/data-container: <dataset name> - <md5hash of dataset archive>

Archive Preparation

The command used to prepare archives:

tar -cf - clinical-trials-data-800k  | xz -T 9 -9 -c - > clinical-trials-data-800k.tar.xz

Published Datasets