Cleaner

A toolbox of simple solutions for common data cleaning problems.

Compatible with any Tables.jl implementation.

Installation: At the Julia REPL, using Pkg; Pkg.add("Cleaner")

Key Features

With Cleaner.jl you will be able to:

Format column names to make them unique and fit snake_case or camelCase style.
Remove rows and columns filled with different kinds of empty values. e.g: missing, "", "NA", "None"
Delete columns filled with just a constant value.
Delete rows with at least one missing value.
Use a row as the names of the columns.
Minimize the amount of element types for each column without making the column of type Any.
Add a row index to your table.
Automatically use multiple threads if your data is big enough (and you are running Julia with more than 1 thread).
Rematerialize your original source Tables.jl type, as CleanTable implements the Tables.jl interface too.
Apply Cleaner transformations on your original table implementation and have the resulting table be of the same type as the original.
Get all repeated values or value combinations that are supposed to be unique.
Get the percentage distribution of the different categories that make up your table.
Compare tables to help solve join or merge problems caused by having different schemas.

To keep in mind

All non mutating functions (those ending without a !) receive a table as argument and return a CleanTable.
All mutating functions (those ending with a !) receive a CleanTable and return a CleanTable.
All returning original type function variants (those ending with ROT) receive a table as argument and return a table of the same type of the original.

So you can start your workflow with a non mutating function and continue it using mutating ones if you prefer. E.g.

julia> df = DataFrame(" some bad Name" => [missing, missing, missing], "Another_weird name " => [1, 2, 3])
3×2 DataFrame
 Row │  some bad Name  Another_weird name
     │ Missing         Int64
─────┼─────────────────────────────────────
   1 │        missing                    1
   2 │        missing                    2
   3 │        missing                    3

julia> df |> polish_names |> compact_columns!
┌────────────────────┐
│ another_weird_name │
│              Int64 │
├────────────────────┤
│                  1 │
│                  2 │
│                  3 │
└────────────────────┘

Related Packages

DataFrames.jl for general and complex tabular data transformations.
TableOperations.jl for common lazily evaluated transformations on Tables.jl implementations.
FeatureTransforms.jl, TableTransforms.jl for statistical data transformations on Tables.jl implementations.

Acknowledgement

Inspired by janitor from the R ecosystem.