symflower/eval-dev-quality

Issues

• Assess failing tests

  #235 opened 13 hours ago
  0

• Weight "executed code" more prominently

  #233 opened 13 hours ago
  1

• Follow-up: Apply "symflower fix" to a "write-test" result of a model when it errors, so model responses can possibly be fixed

  #232 opened 13 hours ago
  0

• Make the Knapsack.java case easier to solve for models

  #230 opened 2 days ago
  0

• Do not start the ollama server if not needed

  #225 opened 6 days ago
  0

• Follow up "Docker runtime"

  #224 opened 6 days ago
  0

• Multiple model parameter with same value result in multiple evaluations

  #220 opened 7 days ago
  0

• unable to create temporary repository path: exec: WaitDelay expired before I/O complete

  #219 opened 7 days ago
  2

• Report the maximum theoretically reachable #files-executed

  #215 opened 8 days ago
  0

• Apply symflower fix to a "write-test" result of a model

  #213 opened 13 hours ago
  3

• Extract model costs into log and CSVs

  #210 opened 9 days ago
  1

• Do an evaluation run for all "good open weight models" with all available quantizations and different GPUs

  #209 opened 10 days ago
  0

• Interactive result comparison

  #208 opened 12 days ago
  0

• Add the current commit revision to the binary, Docker image and reports

  #207 opened 12 days ago
  0

• Extract human-readable names for models

  #206 opened 7 days ago
  0

• Tool/command to combine multiple evaluations into one

  #205 opened 13 days ago
  1

• Keep individual coverage files and LLM query/responses

  #204 opened 14 days ago
  2

• Evaluation task: Transpile

  #201 opened 15 days ago
  2

• Follow-up "Code repairing task to enable models to fix code with compilation errors"

  #200 opened 16 days ago
  0

• Isolation of evaluations

  #198 opened 16 days ago
  0

• Roadmap for v0.6.0

  #195 opened 16 days ago
  0

• Evaluation task: TDD

  #194 opened 16 days ago
  0

• Coverage for Java is tracked for lines, while Go is tracked for ranges

  #193 opened 19 days ago
  4

• CSV report header is missing the task identifier

  #187 opened 19 days ago
  0

• Openrouter returns 524 when querying models

  #186 opened 19 days ago
  0

• Add timeout to `symflower test`

  #185 opened 20 days ago
  0

• Log model responses directly to file and reuse them for debugging

  #181 opened 21 days ago
  0

• If results folder already exists, add suffix but don't overwrite or error

  #176 opened 21 days ago
  0

• Collect Go coverage if tests trigger panic

  #175 opened 21 days ago
  1

• Deal with dependencies requested by LLMs

  #174 opened 23 days ago
  2

• LLM result parsing bug

  #173 opened 22 days ago
  3

• Improve maintainability of assessments

  #169 opened 19 days ago
  2

• Evaluation task: Code repair

  #168 opened 15 days ago
  0

• Fixed timeouts for `symflower unit-tests` and `symflower test`

  #167 opened a month ago
  2

• Support multiple evaluation tasks

  #165 opened a month ago
  0

• `ollama_llama_server` and other background processes we start must be killed on CTRL+C

  #164 opened a month ago
  3

• Automatic selection of repositories is broken

  #163 opened 15 days ago
  0

• New task to check for Go and Java compilation errors

  #160 opened 21 days ago
  0

• https://github.com/symflower/eval-dev-quality/pull/155/files missing a test

  #159 opened 16 days ago
  3

• Deal with failing tests

  #158 opened a month ago
  0

• Logic for "Create temporary repositories for each language so the repository is copied only once per language." copies more than needed

  #157 opened 21 days ago
  0

• Running Ollama tests with the wrong Ollama binary should fail hard

  #156 opened 22 days ago
  1

• The prompt uses different paths depending on the OS

  #152 opened a month ago
  0

• Evaluation folder with date cannot be created on windows

  #151 opened a month ago
  0

• Repository not reset for multiple tasks

  #147 opened a month ago
  0

• The git repository change requires the GPG password

  #145 opened a month ago
  0

• Java

  #143 opened a month ago
  0

• Follow-Up from using Git to reset the temporary directory

  #141 opened 20 days ago
  0

• Test for pulling Ollama model is flaky

  #135 opened a month ago
  2

• Follow-up: Allow to retry a model when it errors

  #131 opened a month ago
  0