Long Code Arena is a suite of benchmarks for code-related tasks with large contexts, up to a whole code repository. It currently spans six different tasks and contains six datasets:
- 🤗 Library-based code generation
- 🤗 CI builds repair
- 🤗 Project-level code completion
- 🤗 Commit message generation
- 🤗 Bug localization
- 🤗 Module summarization
For each task, we have different approaches and environments. You can find the baseline solutions and the necessary information about each task in the corresponding directories.
We are excited to invite you to participate in solving our benchmarks! To submit your results, please send the following materials to our 📩 email (lca@jetbrains.com):
- Results: Include the summary of your benchmark outcomes.
- Reproduction Package: To ensure the integrity and reproducibility of your results, please include the code for context collection (if any), generation of predictions, and evaluating. You can follow our baselines as a reference.
- Metadata: Model information, organization name, licence of your model, context size, and other information you find relevant.