auto_ab - automation of A/B-experiments
auto_ab is a library developed to automate routine tasks during runs of multiple online controlled experiments, also known as A/B-tests.
Current features
- Custom metrics
- Splitting
- Splitter
- Test splitter (A/A-test)
- Custom split rate and increment rate
- Continuous metrics
- Classical hypothesis testing
- Bootstrap hypothesis testing with 3 different approaches
- Bucketing hypothesis testing
- Ratio metrics
- Delta method on ratios
- Taylor expansion on ratios
- Linearization on ratios
- Bootstrap on ratios
- Variance reduction
- CUPED
- CUPAC
Config file
Parameters for A/B-test are set in config file. The example of config file you can find below.
metric:
metric_type: 'ratio'
metric_name: 'mean'
data:
id_col: 'id_column_name'
group_col: 'group_column_name'
target: 'target_column_name'
numerator: 'numerator_column_name'
denominator: 'denominator_column_name'
splitter:
run_aa: True
simulation:
n_iter: 100
split_rates: [0.1, 0.2, 0.3, 0.4, 0.5]
increment:
vars: [0, 1, 2, 3, 4, 5]
extra_params: []
hypothesis:
alpha: 0.05
beta: 0.2
alternative: 'two-sided'
split_ratios: [0.5, 0.5]
strategy: 'ttest'
n_boot_samples: 200
variance_reduction:
method: 'cuped'
strata: 'country'
strata_weights:
'US': 0.5
'UK': 0.5
result:
to_csv: True
csv_path: './result.csv'
Config description
parameter | description | values |
---|---|---|
metric_type | Describes metric type | 'solid' for continuous metrics 'ratio' for ratio metric |
metric_name | Describes metric name | 'mean', 'median' |
hypothesis.alpha | Type I error | Must be in [0; 1]. Default is 0.05 |
hypothesis.beta | Type II error | Must be in [0; 1]. Default is 0.2 |
hypothesis.alternative | Hypothesis alternative | 'less', 'greater', 'two-sided' |