neulab/explainaboard_web

Move benchmark configuration to database

Closed this issue · 2 comments

neubig commented

Currently the benchmark is here:
https://github.com/neulab/explainaboard_web/tree/main/backend/src/impl/benchmark_configs

But it'd be nice to have it in the DB so it's easier to play with benchmark configurations without re-deploying the server.

pfliu-nlp commented

I try to revisit this task (cc @OscarWang114 )

So far, benchmark configs are stored as JSON files that will be loaded through this function

To DBlize this process, we may have the following workflow:

  • (1) Create a DB for benchmark configs based on their current schemas
  • (2) Write an API that can post benchmark config file, validate its schema, and store it in DB
  • (3 )Write a very simple UI interface here for users to upload (e.g., create button) their benchmark config files

Hi, @neubig do you have any comments about the above workflow?

OscarWang114 commented

@pfliu-nlp Thanks for detailing the workflow! I think it looks great, and I'm planning to submit 2 PRs: PR1 will contain (1) and (2), and PR2 will contain (3). Basically the first focuses on the DB + backend, and the second focuses on the frontend.