xfeat

Slides | Tutorial | Document | Installation

Flexible Feature Engineering & Exploration Library using GPUs and Optuna.

xfeat provides sklearn-like transformation classes for feature engineering and exploration. Unlike sklearn API, xfeat provides a dataframe-in, dataframe-out interface. xfeat supports both pandas and cuDF dataframes. By using cuDF and CuPy, xfeat can generate features 10 ~ 30 times faster than a naive pandas operation.


Group-by aggregation benchmark (result)	Target encoding benchmark (result)

Document

Slides
Tutorial notebook
Feature Encoding and Pipelining
Target encoding and benchmark result
Group-by aggregation and benchmark result
Feature selection with Optuna

More examples are available in the ./examples directory.

Quick Start

xfeat provides a dataframe-in, dataframe-out interface:

Feature Engineering

It is possible to sequentially concatenate encoder objects with xfeat.Pipeline. To avoid repeating the same feature extraction process, it is useful to output the results to the feather file format.

More encoder classes available here.

import pandas as pd
from xfeat import Pipeline, SelectNumerical, ArithmeticCombinations

# 2-order Arithmetic combinations.
Pipeline(
    [
        SelectNumerical(),
        ArithmeticCombinations(
            exclude_cols=["target"], drop_origin=True, operator="+", r=2,
        ),
    ]
).fit_transform(pd.read_feather("train_test.ftr")).reset_index(
    drop=True
).to_feather(
    "feature_arithmetic_combi2.ftr"
)

Target Encoding with cuDF/CuPy

Target encoding can be greatly accelerated with cuDF. Internally, aggregation is computed on the GPU using CuPy.

from sklearn.model_selection import KFold
from xfeat import TargetEncoder

fold = KFold(n_splits=5, shuffle=False)
encoder = TargetEncoder(input_cols=cols, fold=fold)

df = cudf.from_pandas(df)  # if cuDF is available.
df_encoded = encoder.fit_transform(df)

Groupby features with cuDF

Benchmark result: Group-by aggregation and benchmark result.

from xfeat import aggregation

df = cudf.from_pandas(df)  # if cuDF is available.
df_agg = aggregation(df,
                     group_key="user_id",
                     group_values=["price", "purchased_amount"],
                     agg_methods=["sum", "min", "max"]
                     ).to_pandas()

Feature Selection with GBDT feature importance

Example code: examples/feature_selection_with_gbdt.py

from xfeat import GBDTFeatureSelector

params = {
    "objective": "regression",
    "seed": 111,
}
fit_kwargs = {
    "num_boost_round": 10,
}

selector = GBDTFeatureSelector(
    input_cols=cols,
    target_col="target",
    threshold=0.5,
    lgbm_params=params,
    lgbm_fit_kwargs=fit_kwargs,
)
df_selected = selector.fit_transform(df)
print("Selected columns:", selector._selected_cols)

Feature Selection with Optuna

GBDTFeatureSelector uses a percentile hyperparameter to select features with the highest scores. By using Optuna, we can search for the best value for this hyperparameter to maximize the objective.

Example code: examples/feature_selection_with_gbdt_and_optuna.py

import optuna

def objective(df, selector, trial):
    selector.set_trial(trial)
    selector.fit(df)
    input_cols = selector.get_selected_cols()

    # Evaluate with selected columns
    train_set = lgb.Dataset(df[input_cols], label=df["target"])
    scores = lgb.cv(LGBM_PARAMS, train_set, num_boost_round=100, stratified=False, seed=1)
    rmsle_score = scores["rmse-mean"][-1]
    return rmsle_score


selector = GBDTFeatureExplorer(
    input_cols=input_cols,
    target_col="target",
    fit_once=True,
    threshold_range=(0.6, 1.0),
    lgbm_params=params,
    lgbm_fit_kwargs=fit_params,
)

study = optuna.create_study(direction="minimize")
study.optimize(partial(objective, df_train, selector), n_trials=20)

selector.from_trial(study.best_trial)
print("Selected columns:", selector.get_selected_cols())

Installation

$ python setup.py install

If you want to use GPUs, cuDF and CuPy are required. See the cuDF installation guide.

For Developers

$ python setup.py test

pfnet-research/xfeat