[Question] Can I set/change the traffic allocation between variations?
Closed this issue · 6 comments
@gingerlime Hi I love your simple and awesome AB Testing framework, I have a question can I set/change traffic allocation between variations? I see there's a 'sample size' param which is good,
Cheers,
Thanks
Hey @erwinyusrizal. Thanks. I'm happy you like Alephbet.
The sample size is used only to limit to overall participants in the experiment. Not to control the ratio between variants.
Controlling the ratio between variants is a technique used by multi-armed-bandit algorithms, and it's a direction I'd be happy to explore in general. But there are non-trivial things to consider when implementing it, so I haven't had a chance to tackle it yet.
That said, statically controlling the ratio between variants isn't super complicated on its own, and could make a nice addition. It's not available yet, but shouldn't be too difficult to achieve. If you think this might be something you'd be interested in contributing via a PR, then I'd be happy to review and get this integrated. As long as it doesn't break backwards compatibility of course.
@gingerlime about the sample size, yes it is for participants, which is good. about the ratio between variants I'm will exploring it too.
also it would be good if it has start and stop experiment too at least based on datetime, so we can stop the experiment if we feel it's inconclusive.
again I love your framework
Cheers,
Stopping an experiment is as simple as removing the experiment JS code :) or you can write your trigger or activate functions that would look up some flag, or wrap the whole experiment inside your own callback. e.g.
if (condition_to_run_experiment) {
experiment = AlephBet.Experiment(...);
}So I'm not sure I'd want something built-in into Alephbet for that purpose.
That said, adding some kind of a callback to switch the whole experiment on/off is trivial to implement and wouldn't make things too messy, so I'm just undecided about it rather than dead-against it at this point ...
Yes thats what I did at the moment, thanks for get back to me on it
You're very welcome, @erwinyusrizal.
@erwinyusrizal there's a new weight feature implemented in #20 and released in v0.17.0 :)