/uptasticsearch

An Elasticsearch client tailored to data science workflows.

Primary LanguageR

uptasticsearch

Introduction

This project tackles the issue of getting data out of Elasticsearch and into a tabular format in R.

Table of contents

  1. How it Works
  2. Installation
    1. R
    2. Python
  3. Usage Examples
    1. Get a Batch of Documents
    2. Aggregation Results
  4. Next Steps
    1. Auth Support

How it Works

The core functionality of this package is the es_search function. This returns a data.table containing the parsed result of any given query. Note that this includes aggs queries.

Installation

R

The package is not yet on CRAN, but we'll be submitting soon. Installation instructions coming soon.

Python

We plan to release a Python implementation of this functionality, but that is not available at this time. Check back often!

Usage Examples

The examples presented here pertain to a fictional Elasticsearch index holding some information on a movie theater business.

Example 1: Get a Batch of Documents

The most common use case for this package will be the case where you have an ES query and want to get a data frame representation of many resulting documents.

In the example below, we use uptasticsearch to look for all survey results in which customers said their satisfaction was "low" or "very low" and mentioned food in their comments.

    # Build your query in an R string
    qbody <- '{
      "query": {
        "filtered": {
          "filter": {
            "bool": {
              "must": [
                {
                  "exists": {
                    "field": "customer_comments"
                  }
                },
                {
                  "terms": {
                    "overall_satisfaction": ["very low", "low"]
                  }
                }
              ]
            }
          }
        },
        "query": {
            "match_phrase": {
              "customer_comments": "food"
            }
        }
      }
    }'

    # Execute the query, parse into a data.table
    commentDT <- es_search(es_host = 'http://mydb.mycompany.com:9200'
                           , es_index = "survey_results"
                           , query_body = qbody
                           , scroll = "1m"
                           , n_cores = 4)

Example 2: Aggregation Results

Elasticsearch ships with a rich set of aggregations for creating summarized views of your data. uptasticsearch has built-in support for theses aggregations.

In the example below, we use uptasticsearch to create daily timeseries of summary statistics like total revenue and average payment amount.

    # Build your query in an R string
    qbody <- '{
      "query": {
        "filtered": {
          "filter": {
            "bool": {
              "must": [
                {
                  "exists": {
                    "field": "pmt_amount"
                  }
                }
              ]
            }
          }
        }
      },
      "aggs": {
        "timestamp": {
          "date_histogram": {
            "field": "timestamp",
            "interval": "day"
          },
          "aggs": {
            "revenue": {
              "extended_stats": {
                "field": "pmt_amount"
              }
            }
          }
        }
      },
      "size": 0
    }'

    # Execute the query, parse result into a data.table
    revenueDT <- es_search(es_host = 'http://mydb.mycompany.com:9200'
                           , es_index = "transactions"
                           , size = 1000
                           , query_body = qbody
                           , n_cores = 1)

In the example above, we used the date_histogram and extended_stats aggregations. es_search has built-in support for many other aggregations and combinations of aggregations, with more on the way. Please see the table below for the current status of the package. Note that names of the form "agg1 - agg2" refer to the ability to handled aggregations nested inside other aggregations.

Agg type R support?
"cardinality" YES
"date_histogram" YES
date_histogram - cardinality YES
date_histogram - extended_stats YES
date_histogram - histogram YES
date_histogram - percentiles YES
date_histogram - significant_terms YES
date_histogram - stats YES
date_histogram - terms YES
"extended_stats" YES
"histogram" YES
"percentiles" YES
"significant terms" YES
"stats" YES
"terms" YES
terms - cardinality YES
terms - date_histogram YES
terms - date_histogram - cardinality YES
terms - date_histogram - extended_stats YES
terms - date_histogram - histogram YES
terms - date_histogram - percentiles YES
terms - date_histogram - significant_terms YES
terms - date_histogram - stats YES
terms - date_histogram - terms YES
terms - extended_stats YES
terms - histogram YES
terms - percentiles YES
terms - significant_terms YES
terms - stats YES
terms - terms YES
"stats" YES

Next Steps

This is a fairly new project and, as the version number indicates, should be regarded as a work in progress.

Auth Support

uptasticsearch does not currently support queries with authentication. This will be added in future versions.