ExpediaGroup/adaptive-alerting

Clean up the MappedMetricData data schema

Closed this issue · 2 comments

There are several issues:

  • The metric definition appears twice
  • Detector UUID appears twice.
  • Detector UUID appears as detectorUuid one time and detectorUUID the other time.
  • The schema includes the investigationResults field, which we no longer use in AA
  • We reference prediction/threshold related fields that don't make sense for many of the detectors

Whoever takes this issue, let's discuss more carefully what the target data schema should be.

Expedia-specific: We will need to roll this out in a way that minimizes client impact. Some potential coordination areas:

  • Aquila is posting anomalies, so we will need to update it
  • Seyren will be posting anomalies, so we will need to update that
  • Vector is reading anomalies
  • Armor is reading anomalies
  • I don't think Haystack is reading anomalies directly, but double-check

Here's an example:

  {
    "metricData": {
      "metricDefinition": {
        "tags": {
          "kv": {
            "mtype": "gauge",
            "unit": "metric",
            "what": "[redacted]",
            "m_application": "[redacted]",
            "org_id": "1",
            "name": "[redacted]",
            "interval": "60",
            "m_account": "[redacted]",
            "m_region": "us-west-2"
          },
          "v": []
        },
        "meta": {
          "kv": {},
          "v": []
        },
        "key": "[redacted]"
      },
      "value": 0,
      "timestamp": 1542295565
    },
    "detectorUuid": "37ef6be4-dd86-11e8-956a-061bc4b8b7e4",
    "detectorType": "constant-detector",
    "anomalyResult": {
      "detectorUUID": "37ef6be4-dd86-11e8-956a-061bc4b8b7e4",
      "metricData": {
        "metricDefinition": {
          "tags": {
            "kv": {
              "mtype": "gauge",
              "unit": "metric",
              "what": "[redacted]",
              "m_application": "[redacted]",
              "org_id": "1",
              "name": "[redacted]",
              "interval": "60",
              "m_account": "[redacted]",
              "m_region": "us-west-2"
            },
            "v": []
          },
          "meta": {
            "kv": {},
            "v": []
          },
          "key": "[redacted]"
        },
        "value": 0,
        "timestamp": 1542295565
      },
      "anomalyLevel": "NORMAL",
      "predicted": null,
      "thresholds": null,
      "investigationResults": null
    }
  }

Also, this issue gives us a good opportunity to document an approach to schema migration.

Updates

  • Probably MappedMetricData shouldn't have an AnomalyResult at all.
  • Probably we should push AnomalyResult onto the anomaly topic instead of pushing a MappedMetricData onto that topic.

I removed the investigationResults field.