Sample Ratio Mismatch (SRM) Checker

The Sample Ratio Mismatch (SRM) test can be used to detect a wide variety of data quality issues (paper) that may affect online experiments (aka A/B tests). Only expected proportions and observed sample counts are required as input for this procedure, so this test can be used even in cases where experimenters only have access to summary statistics; such as when using third-party tools.

This repository contains a Chrome Extension that can automatically detect SRM on supported platforms, as well as a website with more information about the extension.

Platforms supported by this Chrome Extension:

Convert.com
Google Optimize
Omniconvert
Optimizely
SiteGainer/Symplify
Visual Website Optimizer (VWO)
Zoho PageSense

Several experimentation platforms natively perform Sample Ratio Mismatch checks for their users. These platforms will not be supported by our Chrome Extension, because that would be redundant. An incomplete overview of these platforms can be found here.

Known issues:

Convert: Auto-stopping variants triggers false SRM alert (#29)

About the project

Sample Ratio Mismatch Checker was created by Lukas Vermeer, Heinrich Mahr and Georgi Georgiev.

License

Sample Ratio Mismatch Checker is distributed by an MIT license.

Contributing

You want to contribute to this project? That's amazing! We'd love that! <3

Feel free to fork the repository on on GitHub, make the changes you want to see, and issues a pull request. If you are not sure what to work on, we maintain a list of outstanding issues on the GitHub project. (If you don't want to code, adding to that list of issues is also a great way to contribute.)

Thank you for making this project more awesome.

Testing locally

We utilize the instructions provided here regrading testing locally with Github Pages using Jekyll.

These instructions are only for UNIX-based setups.

Initial, One-time Steps

Make sure chruby and Ruby are installed. Use your package manager or sudo apt-get install ruby-dev or sudo apt-get install ruby-full. On a Mac, these instructions are quite good.
Make sure you are using Ruby version 2.7. Use ruby -v to check which version is currently active.
Install Bundler - gem install bundler.
Install Jekyll via sudo apt-get install ruby-full build-essential zlib1g-dev and gem install jekyll bundler.

Running Locally

All you gotta do is run this command at the root of the repo (where the Gemfile is located):

bundle exec jekyll serve

You will get output that looks like the following

Configuration file: /media/paul/3638-6263/git/srm/_config.yml
            Source: /media/paul/3638-6263/git/srm
       Destination: /media/paul/3638-6263/git/srm/_site
 Incremental build: disabled. Enable with --incremental
      Generating... 
Invalid theme folder: _sass
      Remote Theme: Using theme pmarsceill/just-the-docs
       Jekyll Feed: Generating feed for posts
                    done in 5.583 seconds.
 Auto-regeneration: enabled for '/media/paul/3638-6263/git/srm'
    Server address: http://127.0.0.1:4000/srm/
  Server running... press ctrl-c to stop.

Note the Invalid theme folder: _sass will show up as a warning; you can safely ignore it as it's related to production deployment on Github. Navigate to the server address given (in this case, http://127.0.0.1:4000/srm/ and you're done!