metacall/core

Review Windows Benchmark Issue and possible improvement of gh-pages

viferga opened this issue · 6 comments

viferga commented

🐛 Bug Report

For some reason, nodejs benchmark is failing with an invalid json. We should print the json and see what's wrong with it in order to debug it, or skip the test.

Another improvement is to deploy directly to github pages instead of pushing to a branch:
https://github.com/actions/upload-pages-artifact

Here's an example:

# Sample workflow for building and deploying a Jekyll site to GitHub Pages
name: Deploy Jekyll with GitHub Pages dependencies preinstalled

on:
  # Runs on pushes targeting the default branch
  push:
    branches: ["develop"]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
permissions:
  contents: read
  pages: write
  id-token: write

# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
# However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
concurrency:
  group: "pages"
  cancel-in-progress: false

jobs:
  # Build job
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Setup Pages
        uses: actions/configure-pages@v3
      - name: Build with Jekyll
        uses: actions/jekyll-build-pages@v1
        with:
          source: ./
          destination: ./_site
      - name: Upload artifact
        uses: actions/upload-pages-artifact@v2

  # Deployment job
  deploy:
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    runs-on: ubuntu-latest
    needs: build
    steps:
      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v2
viferga commented

@ahmedihabb2 I have solved the issue of windows, now all benchmarks are passing, it would be good if we can delete the gh-pages branch and just deploy directly to github pages with the methodology I posted.

viferga commented

@ahmedihabb2 I have solved almost everything, there's something left I have noticed. First of all, the URL is not completely correct, right now it is:
https://metacall.github.io/core/windows-2019/

And it should be:
https://metacall.github.io/core/bench/windows-2019/

And there's something else, I think it has no histogram right now, we have broke it because this action was using the commit history of gh-pages for storing it.

So for doing this I have few ideas, publishing the final json of the generated benchmark as artifact (not the one generated by cmake but the one generated by the action) and then pulling it each time, then passing it to the action. Or doing the same with the index.html and index.js of the website.

I am still not 100% sure about this because I have not fully tested it, but it seems so.

I know this may seem overengineering but I do not want to couple the repo branches to the deployment method (aka, having a gh-pages for storing generated data in order to host it). This is because I have multiple origins and I do not want to pollute the repo history with this, it's a bad solution.

Another alternative should be publishing the website into another repo, which was my original idea, but I saw the current method we are using was available now from GitHub. Also setting a personal token for doing push to another repo seems a bit insecure to me.

I leave you this for debugging and finding a solution.

viferga commented

Apparently I was on the right track, check this out:
58b8609

I took it from here:
https://github.com/benchmark-action/github-action-benchmark#minimal-setup

I think it requires to upload the file to the cache but I am too tired right now:

      # Upload the updated cache file for the next job by actions/cache

I leave it to you. Here's the doc of the cache: https://github.com/actions/cache

ahmedihabb2 commented

It turns out cache won't work with us, as benchmark action does not generate index.html and data.js files when the external-data-json-path is enabled
So we agreed to store the benchmark results in another repo dedicated to benchmarks only and host GitHub pages from it
#464 Check this PR

@viferga

viferga commented

Everything is working like a charm now, we have a base at least now, we should keep improving this from time to time. Windows issues will be solved eventually. I am closing this issue for now. Great job @ahmedihabb2

  • ❤️1