Implement tuning db merger
gevtushenko opened this issue · 3 comments
Tuning on multiple nodes or GPUs leads to multiple tuning databases. To facilitate analysis, we should implement a merger that would apply Mann–Whitney U test to base variants across databases and only then merge corresponding variant records into a single database.
What's the purpose of testing with a U-test before merging the database? So long as a given variant is built from the same commit hash with the same compiler and environment, it should be fine to merge, shouldn't it?
@jrhemstad, the idea was that the results achieved on different GPUs might vary. The base versions are run on each GPU while variants are distributed. This means the variant is relative to its base, not to all the bases. To be safe, I wanted to double-check that the base results have a high probability of belonging to a single distribution. For instance, if one of 8 GPUs is 3% faster, the united base distribution will be biased to 3% slower results. If a given variant was executed only on the fast GPU and afterwards compared to the united base distribution, it'll gain ~3% speedup, obscuring other variants.
Ah, of course, that makes sense.