ctuning/artifact-evaluation

Collective Knowledge repository to support artifact evaluation and reproducibility initiatives:

Data license: Code license:

Artifact evaluation

• Website: https://cTuning.org/ae

Maintainers

• Grigori Fursin

Automation

We are developing Collective Mind in collaboration with MLCommons to automate reproducibility initiatives and artifact evaluation:

• GitHub
• ACM REP'23 keynote about "Collective Mind: toward a common language to facilitate reproducible research and technology transfer"
• White paper about "enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments"

License

Data

• CC BY 4.0 license

Code

• Apache 2.0 for the CM framework (2022-cur.)
• BSD 3-clause for the CK framework (2014-2022)

Copyright

2014-2024 cTuning foundation

Public discussions

• Artifact Evaluation google group

Related publications and presentations

• Artifact Evaluation for Publications (Dagstuhl Perspectives Workshop 15452)
• "Community-driven Reviewing and Validation of Publications"
• "Collective Knowledge: organizing research projects as a database ofreusable components and portable workflows with common APIs"
• ACM TechTalk'21 about "Reproducing 150 Research Papers and Testing Them in the Real World: Challenges and Solutions"

Acknowledgments

Partial list (2014-2021) is available here.