Add locale rotation for pluse triggered builds on a daily basis
Closed this issue · 13 comments
As we discussed last week in the Automation meeting there is less likely the need to test all locales on a daily basis for mozilla-aurora. Therefore we might want to select about 20 locales in a random order for tests and let it rotate daily. en-US should always be included. With that we will be able to test each and every locale within a single week.
For beta and release we should test them all in the near future.
@ashughes1 does it sound fine to you?
Regarding mozilla-central we shouldn't run any locale at all. Reason is that l10n development happens nearly only on mozilla-aurora and downwards.
Sorry for the delay (at a work week currently). My question is how much capacity do we have (ie. how many locales at once can we support)? If we can handle 20 I would prefer to have a standard set of 10 that we always test (ie. the ones we test via pulse on beta/release currently) and randomly select 10 more locales from the remaining 80+ locales we support.
I think that we can handle 20 to 30 locales. If we could take 30 locales which ratio would you prefer? Means how many static locales compared to randomly chosen ones?
In the short term lets not worry too much about ratio. Why not just keep the standard set as it is and add as much randomized capacity as you can handle? We can always tweak as necessary in the future.
As long as you don't say we have to definitely test specific locales each day, I'm ok with that. That's why I even proposed to have en-US static and all others random. So please let me know which locales should definitely be static.
Yeah, let's just go with en-US static + N random locales (for whatever value of N the infrastructure can handle). This should be fine as long as you can ensure every locale is tested at least once per 6-week cycle.
So given all the feedback here we can do the following:
-
For aurora builds we take the en-US build each day and add N (I assume around 20) locales to the list. We will store which locales we ran, so we do not take the same ones the next day. We iterate through all of them before starting from the beginning. With about 90 locales I expect a turnaround time of about 5 days. Tests we will execute are update, functional, remote, and add-on and endurance for en-US only.
-
For beta and release builds we will most likely take all actively supported locales and run them each time for the following tests: functional, remote, and add-on and endurance for en-US only.
@ashughes1 and @davehunt how does this sound?
Sounds fine to me.
Sounds fine to me as well. Just to confirm...
Nightly: update/functional/remote/add-on/endurance (en-US)
Aurora: update/functional/remote (en-US & 20 l10n), add-on/endurance (en-US)
Beta: update/functional/remote (en-US & all l10n), add-on/endurance (en-US)
Release: update/functional/remote (en-US & all l10n), add-on/endurance (en-US)
For beta and release we do not run update tests. Those have still to be done via ondemand. So here the corrected list:
Nightly: update/functional/remote/add-on/endurance (en-US)
Aurora: update/functional/remote (en-US & 20 l10n), add-on/endurance (en-US)
Beta: functional/remote (en-US & all l10n), add-on/endurance (en-US)
Release: functional/remote (en-US & all l10n), add-on/endurance (en-US)
Okay, yeah that makes sense to me.
For beta and release we do not run update tests. Those have still to be done via ondemand.
I'd like it if we could get to a point where update tests are kicked off automatically and the person driving the release only needs to triage the results, but I suppose that's a different bug.
At any rate, I'm okay with these changes as proposed.
I'd like it if we could get to a point where update tests are kicked off automatically and the person driving the release only needs to triage the results, but I suppose that's a different bug.
That's clearly another issue we will have to work on. But as you say, it fits into another issue. Feel free to raise it with your specific needs. We can then figure out how to do that.
I'm not sure when I will have time to get this feature here implemented, so I will also open it as a mentored issue.
That's clearly another issue we will have to work on.
Raised in issue #409.