meandavejustice/min-vid

Performance Testing

Standard8 opened this issue · 13 comments

Standard8 commented

The basic summary of what to do is:

  • Create a patch to insert the add-on into mozilla-central (see below)
  • Do two try builds. The first is without the patch, the second is with the patch. They should be based off the same revision of mozilla-central. Use a try chooser command such as try: -b o -p linux64,linux64-pgo,macosx64,win32,win32-pgo,win64,win64-pgo -u none -t all --artifact --rebuild 6 (not you need at least 6 rebuilds for enough comparisons, you may need a few retriggers as well). You should also be able to get away with artifact builds.
  • At some stage you'll get an option to build a link, if not you can use this link and just update the revisions for it.

Once the results come in, they'll appear on the comparison page, and you'll want to look for any red or yellow changes.

For building into mozilla-central. The easiest way is to copy what happens in one of the browser/extensions/ folders of the system add-ons - probably the screenshots one.

  • Add a directory, and add it to browser/extensions/moz.build
  • Copy the add-on files into that directory so that install.rdf/bootstrap.js are in that folder.
  • Then take a look at screenshot's moz.build and adapt that for your files.
  • You should then be able to build & run Firefox, and the add-on should be present.

If you need more help or have questions, feel free to ping me.

jaredhirsch commented

I can take this. I did all this stuff for Screenshots.

  • 🎉1
jaredhirsch commented

Oh no! I applied these patches on top of mozilla-central, but min-vid doesn't work in Nightly.

I'll have to figure out which commit / branch is the correct base for current release channel, then re-run the Talos tests there :-\

jaredhirsch commented

Re-running Talos tests on top of the FIREFOX_59_0_RELEASE tag. mach couldn't find artifacts for a commit this old, so I'm modifying the try incantation to remove the --artifact argument:

try: -b o -p linux64,linux64-pgo,macosx64,win32,win32-pgo,win64,win64-pgo -u none -t all --rebuild 6

Base commit: https://treeherder.mozilla.org/#/jobs?repo=try&revision=febd1a18cbce90f99f95f303c379ba0dff7abb64
Min-Vid without startup delay: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b80aff44b15dc684202645eb9f92b616bb691903
Min-Vid with startup delay: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a649cfd06b617538dcb3db023486ff41be5c8f7c

jaredhirsch commented

Splitting the problem space to try to understand the source of the regressions:

jaredhirsch commented

OK, starting to narrow down a little.

Something either in the StudyUtils.jsm code, or in our use of that code, seems to be causing a lot of the performance problems.

Here's a screenshot of the tp5o regressions on Firefox 59 + minvid + startup delay (results page):

image

If the code that loads and uses StudyUtils.jsm is removed, there are fewer reds, but still a lot of oranges (results page):

image

More detail below, for later reference.

Today, I kicked off Talos runs testing three different changes on both minvid+59 and minvid+60:

  1. effectively disable the whole addon: make startup() and shutdown() do nothing inside bootstrap.js
  2. disable most of the webextension code: remove the background and content_scripts entries from the webextension manifest (so that no code is loaded into the background page & no URL filtering happens)
  3. just remove StudyUtils-related code (partly because I have no idea what this code does; it's minified)

Here are the corresponding Try runs:

minvid-on-59

Talos run against just 59, no minvid: https://treeherder.mozilla.org/#/jobs?repo=try&revision=febd1a18cbce90f99f95f303c379ba0dff7abb64

  1. startup/shutdown no-op: https://treeherder.mozilla.org/#/jobs?repo=try&revision=523621d888f1d3b7f4c892a4958c1b1e9a199b3d
  2. webextension no-op: https://treeherder.mozilla.org/#/jobs?repo=try&revision=80c41a123dd41ee6d7d8ebf531ddb85fdf00405a
  3. StudyUtils removed: https://treeherder.mozilla.org/#/jobs?repo=try&revision=2fff2670382ac98add5a04e264c64eea6fca9c6b

minvid-on-60

Talos run against just 60, no minvid: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ed1ceac6daf75a270cc96511e5dffbbf3a1c48ea

  1. startup/shutdown no-op: https://treeherder.mozilla.org/#/jobs?repo=try&revision=05db4e59614ce6254e46d6cb30e0f4244102efc9
  2. webextension no-op: https://treeherder.mozilla.org/#/jobs?repo=try&revision=80c41a123dd41ee6d7d8ebf531ddb85fdf00405a
  3. StudyUtils removed: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8529a0c5b38a04315c83c8bdace17be9d73cc424
jaredhirsch commented

The last run against 60 produced some inexplicable performance improvements, so I'm re-running everything now: