Dynamic per tag api key support for signalfx sink

[szabado]

Hi!

The intended use case for this is having a per-service API Key, without having to hard code them all in the config file. I'd be adding this to the SignalFX sink.

This idea is @prudhvi's, and I'd be working with him to handle the meat of the implementation. He said he'd chatted with someone and that he'd gotten affirmation that a PR with this logic would be accepted, if it was generic enough.

I had a rough design in mind, let me know if it sounds reasonable. In a ticker polling loop:

• Pull all the access tokens down from https://developers.signalfx.com/org_tokens_reference.html#tag/Retrieve-Tokens-Using-Query
• Hook those into the existing clientsByTagValue map in the signalfx sink.
  • I'm unsure if it should overwrite any tokens specified in the config, or if this should be another layer on top of that (e.g. dynamicClientsByTagValue). I was hoping for some input on that, as I can see arguments either way.

[prudhvi]

cc @aditya-stripe

[prudhvi]

cc @asf-stripe , Any thoughts?

[ChimeraCoder]

Sorry for the delay - I was OOO last week.

This sounds reasonable - it sounds like it'd be simpler to have this override the tokens specified in the config, rather than introducing another set of config options, but I'm open to hearing arguments for doing it that way instead.

[szabado]

The one semi-reasonable reason I had suggested to me for having tokens specified in the config not being overridden: if you wanted to group different apps together under the same token, the manual config could act as an override?

It feels more expected to have the ones fetched from signalfx override the ones specified in the config.