Add experiment YAML linter?

[pdehaan]

GitHub is telling me this may be possibly related to #1308. Fancy!

But I wrote a crude ./content-src/experiments/*.yaml linter at
https://gist.github.com/pdehaan/e61662b050564ee9ec0963f0d7547f04, although it could use a bit of refining.

But so far it caught a few issues w/ a new experiment (#3913 (comment)), as well as potential issues w/ Side View's Facebook and Twitter icon metadata (#3918) and Firefox Color Facebook and Twitter icon metadata (#3917).

I thought we already had a schema linter, but maybe I'm thinking of the theme linter for Firefox Color.

Currently my 💩🔥 proof of concept linter accepts a glob of file(s) and returns errors, but not sure if it'd make sense to have it as a CLI tool as well, or a separate module altogether so you could use it as a bin script, and connect it to a precommit linting script so any new ./content-src/experiments/*.yaml file will be linted before committing. Or, just lint everything in CI.

And finally, here's the output from my script against the price-scout PR branch:

$ node experiment-lint.js

Activity Stream
  checking images:
  checking urls:
    - https://github.com/mozilla/activity-stream/blob/master/CHANGELOG.md { status: 'dead', statusCode: 404 }
    - https://github.com/mozilla/activity-stream/blob/master/docs/v1-test-pilot/data_dictionary.md { status: 'dead', statusCode: 404 }
    - https://testpilot.firefox.com/files/activitystream/latest { status: 'dead', statusCode: 404 }

Advance
  checking images:
  checking urls:

Cliqz
  checking images:
  checking urls:

Color
  checking images:
    - Missing /static/images/experiments/notes/color-fb.jpg
    - Missing /static/images/experiments/notes/color-twitter.jpg
  checking urls:

Containers
  checking images:
  checking urls:

Dev Example
  checking images:
  checking urls:

Firefox Lockbox
  checking images:
  checking urls:

Min Vid
  checking images:
  checking urls:

No More 404s
  checking images:
  checking urls:
    - https://testpilot.firefox.com/files/nomore404s/latest { status: 'dead', statusCode: 404 }

Notes
  checking images:
  checking urls:

Page Shot
  checking images:
  checking urls:
    - https://testpilot.firefox.com/files/pageshot/latest { status: 'dead', statusCode: 404 }

Price Wise
  checking images:
    - Missing /static/images/experiments/price-wise/social/price-wise-facebook.png
    - Missing /static/images/experiments/price-wise/social/price-wise-twitter.png
  checking urls:
    - https://github.com/mozilla/price-wise/blob/master/metrics.md { status: 'dead', statusCode: 404 }

Pulse
  checking images:
  checking urls:
    - https://github.com/mozilla/pulse/issues/new { status: 'dead', statusCode: 404 }

Send
  checking images:
  checking urls:

Side View
  checking images:
    - Missing /static/images/experiments/notes/side-view-fb.jpg
    - Missing /static/images/experiments/notes/side-view-twitter.jpg
  checking urls:

Snooze Tabs
  checking images:
  checking urls:

Tab Center
  checking images:
  checking urls:
    - https://testpilot.firefox.com/files/tabcentertest1@mozilla.com/latest { status: 'dead', statusCode: 404 }

Tracking Protection
  checking images:
  checking urls:
    - https://testpilot.firefox.com/files/blok/latest { status: 'dead', statusCode: 404 }

Universal Search
  checking images:
  checking urls:

Voice Fill
  checking images:
  checking urls:


$ echo $? # 0

It looks like a bunch of graduated experiments are 404ing on their old xpi_url URLs, so I probably need better logic/handling around that:

https://github.com/mozilla/testpilot/blob/master/content-src/experiments/activity-stream.yaml#L20