/duckplyr_demo

Primary LanguageRMIT LicenseMIT

duckplyr_demo

Companion repository for the duckplyr R package.

Getting Started

First setup and install the libraries and unzip the parquet data

## Install package and dependencies
# install.packages("pak", repos = sprintf("https://r-lib.github.io/p/pak/stable/%s/%s/%s", .Platform$pkgType, R.Version()$os, R.Version()$arch))
pak::pak(c("duckdblabs/duckplyr", "curl", "zip", "tidyverse"))

## Download and unzip data (1.7 GB)
curl::curl_download("http://duckplyr-demo-taxi-data.s3-website-eu-west-1.amazonaws.com/taxi-data-2019-partitioned.zip", "taxi-data-2019-partitioned.zip", quiet = FALSE)
zip::unzip("taxi-data-2019-partitioned.zip")

Running the queries/scripts

To run all duckplyr queries at once run

Rscript duckplyr/run_all_queries.R

To run all dplyr queries at once run

Rscript dplyr/run_all_queries.R

To run just one duckplyr query run

Rscript duckplyr/q0*_**.R

To run just one dplyr query run

Rscript dplyr/q0*_**.R

What do the queries show/highlight?

  1. Highlights duckplyr handling of many small groups

    • Get median tips by day & hour.
    • 168 small groups.
    • Utilizes Perfect hash groups

  2. Highlights duckplyr projection pushdown

    • Gets median tip by the number of passengers
    • explain output shows only total_amount, passenger_count, tip_amount, and month are read from the parquet file.

  3. Highlights duckplyr filter pushdown.

    • Gets popular (pickup, drop-off) combinations in Manhattan.
    • DuckDB can push the filter (Borough = “Manhattan”) all the way into the parquet scan of the dimension table

  4. Highlights duckplyr lazy evaluation.

    • Gets percentage of trips that report no tip. Grouped by (pickup borough, drop-off borogh), ranked by number of trips.
    • Need to join 2 intermediate results,
    • duckplyr lazily evaluates.

  5. Highlights that duckplyr can read hive partitioned data over the network easy. (dplyr cannot do this)

    • Hive partition filters
    • Month filter not in explain output (yet)