Automate results spreadsheet generation for Training v2.1

[erichan1]

This is beyond the scope of just the logging repo, but will put this here as a central place to record requirements for results spreadsheet automation.

@pgmpablo157321 @ckt624 @xyhuang

Requirements:
Automate spreadsheet generation

• Create a github action
• Generate results for each PR to submission repo using results summarizer
• Insert results to a spreadsheet (google sheets?) or display them on github through markdown table.
• If gsheet, have to think about permissions (make a single gsheet, set up the allowed emails, and make new tabs on it every time)

Nice to haves:

• Generate diff between PR branch and main branch results spreadsheet. This would show up in the form of two additional tables. One with lines deleted from main, and one with lines that are new relative to main.
• Add a change to result summarizer so it automatically sorts the outputted results so the final table is sorted (I currently do this manually on google sheets through their sort function).

[erichan1]

My current plan:

1. Upon creation of the v2.1 submissions github repo, we immediately make an empty branch called RESULTS_BRANCH or something similar.
2. We set up github actions so that when pull request id_x comes in, it creates x_results.csv and x_results.md. It puts those files into RESULTS_BRANCH/x_results_folder. Thus, as people make PRs, we make commits only to RESULTS_BRANCH, so we don't clutter main branch.
3. In each submitter PR, we give them a url link to RESULTS_BRANCH/x_results_folder/x_results.md file. We can potentially do this per commit if it's easier.

[erichan1]

Markdown tables display like this.

First Header	Second Header	Second Header	Second Header	Second Header	Second Header	Second Header	Second Header	Second Header	Second Header	Second Header
Content Cell	Content Cell	Content Cell	Content Cell	Content Cell	Content Cell	Content Cell	Content Cell	Content Cell	Content Cell	Content Cell

Source text to get above nice table. Can pretty easily convert a csv to this.

| First Header  | Second Header | Second Header | Second Header | Second Header | Second Header | Second Header | Second Header | Second Header | Second Header | Second Header |
| ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |
| Content Cell  | Content Cell  | Content Cell  | Content Cell  | Content Cell  | Content Cell  | Content Cell  | Content Cell  | Content Cell  | Content Cell  | Content Cell  |

[erichan1]

Another option is to directly comment the results table table into each PR. I originally eliminated this as an option, because the huge table would clutter the PR. But apparently you can have a tooltip.

HEADER

First Header	Second Header	Second Header	Second Header	Second Header	Second Header	Second Header	Second Header	Second Header	Second Header	Second Header
Content Cell	Content Cell	Content Cell	Content Cell	Content Cell	Content Cell	Content Cell	Content Cell	Content Cell	Content Cell	Content Cell

<details> <summary>HEADER</summary> 

| First Header  | Second Header | Second Header | Second Header | Second Header | Second Header | Second Header | Second Header | Second Header | Second Header | Second Header |
| ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |
| Content Cell  | Content Cell  | Content Cell  | Content Cell  | Content Cell  | Content Cell  | Content Cell  | Content Cell  | Content Cell  | Content Cell  | Content Cell  |
</details>

[erichan1]

Here is the code I use to get the final summary csv. Github doesn't like code files, so changed extensions to txt. Change them back. run_summary.sh is the starting point.

summarizer_summarizer.py summarizer_summarizer.txt

run_summary.sh
run_summary.txt

This also required that I have both the training submission repo and the mlperf logging repo cloned to local.

[erichan1]

@pgmpablo157321 Test repo here works perfectly! This already works great and covers the basic requirements. I'd be happy with this without changes.

Two additional nice to haves and then we can call it done:

1. Sort all the outputted CSV files alphabetically/numerically. Sort by all rows, but rows more towards the left in the CSV file have greater priority. This will allow all the closed division results to be together, all the {submitter X} results to be together, etc.
2. Drop the markdown files and output everything as CSVs instead. I didn't know Github will display CSV files nicely, but they do, like here. It's actually nicer than the markdown file because it's more zoomed out. You can also easily copy the raw contents of the csv into a google sheet. So concretely, output summary, added_rows, and removed_rows just as csv files into the folder.

Edit - An even less important ask than the above 2:
3. Add the commit hash(es) to the name of the file. So summary_commithashX.csv, added_rows_commithashX_commithashY.csv. I'm not 100% sure about this one bc it also depends on the current workflow - needs some more discussion.

[pgmpablo157321]

@erichan1

1. Should we take into account the availability column? It is the second column, I don't think that column is as important as the submitter column. For example, here. I think we could swap availability and submitter. What do you think?

[erichan1]

Let's not change the ordering of the columns. As you can see in the final results here https://mlcommons.org/en/training-normal-20/, we group by "Available cloud" or similar.

Also, what is the behavior right now if you add new commits to the same PR?

[pgmpablo157321]

If you add new commits to the PR the workflow will run again and all the spreadsheets will be recomputed

[erichan1]

Sounds good! I'll do some more involved testing on your test_repo soon. But I think this covers everything I can think of.