/ratatool

A tool for random data sampling and generation

Primary LanguageScalaApache License 2.0Apache-2.0

Ratatool

Build Status codecov.io GitHub license Maven Central

A tool for random data sampling and generation

Features

  • Generators - random data generators for Avro, Protocol Buffers and BigQuery TableRow
  • IO - utilities for reading and writing records in Avro, Parquet (via Avro GenericRecord), BigQuery and TableRow JSON files. Local file system, HDFS and Google Cloud Storage are supported.
  • Samplers - random data samplers for Avro, BigQuery and Parquet. True random sampling is supported for Avro only while head mode (sampling from the start) is supported for all sources.
  • ScalaCheck - ScalaCheck generators (Gen[T]) for property-based testing.
  • Diffy - field-level record diff tool for Avro, Protobuf and BigQuery TableRow.
  • BigDiffy - Scio library for pairwise field-level statistical diff of data sets. See slides for more.
  • Command line tool - command line tool for sampling from various sources.

Usage

If you use sbt add the following dependency to your build file:

libraryDependencies += "com.spotify" %% "ratatool" % "0.1.9" % "test"

Or install via our Homebrew tap if you're on a Mac:

brew tap spotify/public
brew install ratatool

Or download the release jar and run it.

wget https://github.com/spotify/ratatool/releases/download/v0.1.9/ratatool-0.1.9.jar
java -jar ratatool-0.1.9.jar

The command line tool can be used to sample from local file system or Google Cloud Storage directly if Google Cloud SDK is installed and authenticated.

java -jar ratatool-0.1.4.jar avro --head -n 1000 --in gs://path/to/dataset --out out.avro
java -jar ratatool-0.1.4.jar parquet --head -n 1000 --in gs://path/to/dataset --out out.parquet

# write output to both JSON file and BigQuery table
java -jar ratatool-0.1.4.jar bigquery --head -n 1000 --in project_id:dataset_id.table_id \
    --out out.json--tableOut project_id:dataset_id.table_id

It can also be used to sample from HDFS with if core-site.xml and hdfs-site.xml are available.

java -cp ratatool-0.1.4.jar:/path/to/hadoop/conf com.spotify.ratatool.tool.Tool avro \
    --head -n 10 --in hdfs://namenode/path/to/dataset --out file:///path/to/out.avro

License

Copyright 2016-2017 Spotify AB.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0