Add Telemetry

[ethan-tonic]

We are adding in very basic telemetry to help us get an idea of what users want in the product. For privacy reasons, we are rolling our own telemetry solution instead of using existing solutions. Only the following information will be logged by the telemetry

• What metrics were used for a run
• Number of questions in a run
• Number of questions in a benchmark

We will NOT track things such as the contents of the questions / answers, scores, or any other sensitive information. We will only track the list of metrics and the number of questions/benchmarks.

[akamor]

I think for logging metrics we should only log if our base metrics are being used. If the customer creates their own metric I think its OK for us to know but let's ensure we are not logging the actual name of their metric as that might be personal.

So the telemetry payload would look something like:

{"base_metrics_used":["AnswerSimilarity","AugmentationPrecision"], "num_custom_metric": 3}

This way we aren't seeing custom metric names but know they are in-use.

[akamor]

Alternatively, we could hash the custom metrics and put them in their own list. This might be more helpful.

e.g.,

{"base_metrics_used":["AnswerSimilarity","AugmentationPrecision"], "custom_metrics_used": ["hash1","hash2"]}