angles-n-daemons/scrutinize

Remove strict mapping between Experiment and Measurement

Closed this issue · 1 comments

I'd like to tie Measurements with multiple treatments, so that metrics can be used as Evaluation Criterion for multiple experiments.

Right now if you do experiment.observe(user_id, 'checked_out', 1) you'll only add a link to the treatment associated with the experiment in question. I'd like to record observations for all experiments associated with the user id. The following needs to happen for this, and the changes are breaking.

  1. Modify the Measurement schema to no longer have a TreatmentID & Variant
  2. Create a new Relation Observation that binds Measurements to Treatments (many to many)
  3. Modify the API for creating measurements to have a "Lookback Period" parameter. This parameter will be a duration in seconds specifying how far back from the current moment to look for treatments to bind to the incoming measurement.
  4. Modify the clients to use the new API.
  5. Modify the reporting tool to query using this new schema.

###Open Question
Should the measurement table be denormalized?

Pro:

  • Performance will be increased if the table is normalized (important for analytical functions).

Con:

  • If there's no treatment to bind, measurements could be dropped.
  • Performance will be dropped, normalizing this will require 2 table joins with a lot of rows

- Old relational model
image

this done man