Add configurable alert annotations?

[benclapp]

Hey! love the project, has been super handy with us getting SLOs rolled out!

Something we did come across when adding was some pain customizing the generated alerts. We can add additional annotations and labels to the generated alerts with the magic of jsonnet.

  prometheusAlerts+:: { groups+: [{
    name: 'some group',
    rules:
      latencyBurn.recordingrules +
      std.map(
        function(alert)
          alert {
            annotations+: { dashboard_url: 'https://grafana.example.com/foo/bar' },
            labels+: { foo: 'bar' },
          },
        latencyBurn.alerts
      ),
  }] },

However I wonder if it may be easier if there was a way to pass in an object of annotations to be added to an alert? Could allow for easier integration into peoples various alert routing setups. Perhaps something like this?

  local query = slo.latencyburn({
    alertName: 'LatencyBudgetBurn',
    metric: 'http_request_duration_seconds',
    selectors: ['namespace="default"', 'job="fooapp"'],
    annotations: {
      dashboard_url: 'https://grafana.example.com/foo/bar',
      message: 'A custom message that overrides the default. May conflict with `alertMessage` in SLOs like errorburn, however.',
    },
    labels: { foo: 'bar' },
    // How much responce delay is too much.
    latencyTarget: '1',
    // The 30 days SLO promise.
    // When the promise is 99% that means that
    // in 30d can only have 1% queries above the latencyTarget.
    latencyBudget: 1 - 0.99,
  }),

Happy to PR in the change if you like, just thought I'd start a discussion before submitting a PR 😄