Fix CI Workflow for Benchmark Execution and Result Handling
tusharmath opened this issue · 25 comments
Description
The CI workflow requires enhancements to handle benchmark execution and updating the README file correctly. We need to address the following scenarios:
- Main Branch: When changes are pushed to the main branch, the CI should run benchmarks and update the README file with the results, committing these changes back to the main branch.
- Pull Requests: For pull requests, the CI should run benchmarks but not update the README or commit any changes. This step is to ensure benchmarks run successfully without altering the repository.
- Failure Handling: The CI should fail if it encounters any issues in generating or updating benchmark results, making these failures visible for prompt resolution.
Expected Behavior
-
On Main Branch:
- Run benchmarks.
- If benchmarks run successfully, update the README file with the results and commit these changes to the main branch.
- If there is a failure in running benchmarks or updating the README, the CI should fail and report the error.
-
On Pull Requests:
- Run benchmarks without updating the README or committing any changes.
- CI should fail if there is an issue in generating benchmark results.
Acceptance Criteria
- CI updates to run benchmarks on the main branch and update the README file with the results, followed by committing these changes.
- CI updates to run benchmarks on pull requests without updating the README or committing any changes.
- Ensure CI fails and reports errors if benchmark generation or README update fails.
/bounty 150$
💎 $150 bounty created by tusharmath
🙋 If you'd like to work on this issue, comment below to get assigned
👉 To claim this bounty, submit a pull request that includes the text /claim #42 somewhere in its body
📝 Before proceeding, please make sure you can receive payouts in your country
💵 Payment arrives in your account 2-5 days after the bounty is rewarded
💯 You keep 100% of the bounty award
🙏 Thank you for contributing to tailcallhq/graphql-benchmarks!
🙋♂️ Join our discord channel if you need help.
/attempt #42
@rishi-raj-jain: The Tailcall Inc. team prefers to assign a single contributor to the issue rather than let anyone attempt it right away. We recommend waiting for a confirmation from a member before getting started.
@tusharmath can you assign this to me? I'd like to work on it.
@Min2who: The Tailcall Inc. team prefers to assign a single contributor to the issue rather than let anyone attempt it right away. We recommend waiting for a confirmation from a member before getting started.
Hey guys @Min2who @rishi-raj-jain, can I take up this one? I've been working on this since the beginning, so I already have all the context/permissions. Hope you don't mind 😄
/attempt #42
@ologbonowiwi: The Tailcall Inc. team prefers to assign a single contributor to the issue rather than let anyone attempt it right away. We recommend waiting for a confirmation from a member before getting started.
@ologbonowiwi go ahead
@ologbonowiwi i have already worked on many part of it should i make a draft pr for you
If you have a near-ready solution, open a PR, and I can help review it.
wait i am making a draft pr for it
It is not really near ready !
should i make a draft pr?
I'm making minor tweaks on tailcallhq/tailcall#662 and tailcallhq/tailcall#640, so I'll not work on it right now, maybe in a few hours. If you can make it in a few hours, feel free to do so then
Hey, @Min2who, I finished the other stuff, so I'm taking it up now.
Hey @tusharmath, I've fixed it successfully.
It ran successfully on main (e.g., https://github.com/tailcallhq/graphql-benchmarks/actions/runs/6942865468/job/18886726403)
and also on #44 (e.g., https://github.com/tailcallhq/graphql-benchmarks/actions/runs/6942885306/job/18886786131?pr=44).
Can you /tip $50 @Min2who and tip the rest to me? As I've committed the changes directly to the main branch, I'll not be able to /claim the bounty.
The solution was to add a Personal Access Token of @tailcall-bot on the repo secrets and add it to the action running.
Feel free to close this one.
@ologbonowiwi The results should be printed in the PR as a separate step. This will allow us to just expand on that step and view results.
I'll create a PR to this then
/tip 50$ @Min2who
/tip 100$ @ologbonowiwi
🎉🎈 @Min2who has been awarded $50! 🎈🎊
@ologbonowiwi: You just got a $50 tip! We'll notify you once it is processed.
/tip 50$ @ologbonowiwi
@ologbonowiwi: You just got a $50 tip! We'll notify you once it is processed.
🎉🎈 @ologbonowiwi has been awarded $50! 🎈🎊
🎉🎈 @ologbonowiwi has been awarded $50! 🎈🎊