data.validator

Description

data.validator is a package for scalable and reproducible data validation. It provides:

Functions for validating datasets in %>% pipelines: validate_if, validate_cols and validate_rows
Predicate functions from assertr package, like in_set, within_bounds, etc.
Functions for creating user-friendly reports that you can send to email, store in logs folder, or generate automatically with RStudio Connect.

Installation

Install from CRAN:

install.packages("data.validator")

or the latest development version:

remotes::install_github("Appsilon/data.validator")

Data validation

Validaton cycle is simple:

Create report object.
Prepare your dataset. You can load it, preprocess and then run validate() pipeline.
Validate your datasets.
- Start validation block with validate() function. It adds new section to the report.
- Use validate_* functions and predicates to validate the data. You can create your custom predicates. See between() example.
- Add assertion results to the report with add_results()
Print the results or generate HTML report.

library(assertr)
library(magrittr)
library(data.validator)

report <- data_validation_report()

validate(mtcars, name = "Verifying cars dataset") %>%
  validate_if(drat < 0, description = "Column drat has only positive values") %>%
  validate_cols(in_set(c(0, 2)), vs, am, description = "vs and am values equal 0 or 2 only") %>%
  validate_cols(within_n_sds(1), mpg, description = "mpg within 1 sds") %>%
  validate_rows(num_row_NAs, within_bounds(0, 2), vs, am, mpg, description = "not too many NAs in rows") %>%
  validate_rows(maha_dist, within_n_mads(10), everything(), description = "maha dist within 10 mads") %>%
  add_results(report)

between <- function(a, b) {
  function(x) { a <= x && x <= b }
}

validate(iris, name = "Verifying flower dataset") %>%
  validate_if(Sepal.Length > 0, description = "Sepal length is greater than 0") %>%
  validate_cols(between(0, 4), Sepal.Width, description = "Sepal width is between 0 and 4") %>%
  add_results(report)

print(report)

Reporting

Print results to the console:

print(report)

# Validation summary: 
#  Number of successful validations: 1
#  Number of failed validations: 4
#  Number of validations with warnings: 1
#
# Advanced view: 
#  
# |table_name |description                                       |type    | total_violations|
# |:----------|:-------------------------------------------------|:-------|----------------:|
# |mtcars     |Column drat has only positive values              |success |               NA|
# |mtcars     |Column drat has only values larger than 3         |error   |                4|
# |mtcars     |Each row sum for am:vs columns is less or equal 1 |error   |                7|
# |mtcars     |For wt and qsec we have: abs(col) < 2 * sd(col)   |error   |                4|
# |mtcars     |vs and am values equal 0 or 2 only                |error   |               27|
# |mtcars     |vs and am values should equal 3 or 4              |warning |               24|

Save as HTML report

save_report(report)

Full examples

Custom reporting on leaflet map

Using custom report templates

In order to generate rmarkdown report data.validator uses predefined report template. You may find it in inst/rmarkdown/templates/standard/skeleton/skeleton.Rmd.

The report contains basic requirements for each report template used by save_report function:

defining params

params:
  generate_report_html: !expr function(...) {}
  extra_params: list()

calling content renderer chunk

```{r generate_report, echo = FALSE}
params$generate_report_html(params$extra_params)
```

If you want to use the template as a base you can use RStudio. Load the package and use File -> New File -> R Markdown -> From template -> Simple structure for HTML report summary. Then modify the template adding custom title, or graphics with leaving the below points unchanged and specify the path inside save_report's template parameter.

How the package can be used in production?

The package was successfuly used by Appsilon in production enviroment for protecting Shiny Apps against beeing run on incorrect data.

The workflow was based on the below steps:

Running RStudio Connect Scheduler daily.
Scheduler sources the data from PostgreSQL table and validates it based on predefined rules.
Based on validation results a new data.validator report is created.

4a. When data is violated:

data provider and person responsible for data quality receives report via email
thanks to assertr functionality, the report is easily understandable both for technical, and non-technical person
data provider makes required data fixes

4b. When data is correct:

a specific trigger is sent in order to reload Shiny data

The workflow is presented on below graphics

Contributing

We welcome contributions of all types!

We encourage typo corrections, bug reports, bug fixes and feature requests. Feedback on the clarity of the documentation and examples is especially valuable.

If you want to contribute to this project please submit a regular PR, once you’re done with new feature or bug fix.

Changes in documentation

Both repository README.md file and an official documentation page are generated with markdown.

Documentation is rendered with pkgdown. Just run pkgdown::build_site() after rendering new README.md.

Appsilon

Appsilon is the Full Service Certified RStudio Partner. Learn more at appsilon.com.

Get in touch support+opensource@appsilon.com

ChensStudio/data.validator