aai-institute/nnbench

Issues

• macOS test failure due to GitHub Apple Silicon oddities

  #145 opened 3 months ago by maxmynter
  0

• Explore a CLI workflow for usage in ML pipelines and CI

  #143 opened 7 months ago by nicholasjng
  0

• Stabilize the record IO interface

  #142 opened 7 months ago by nicholasjng
  0

• Add small parameter representation guide for the benchmark runner

  #141 opened 7 months ago by nicholasjng
  0

• Add `transform` slot to record IO types

  #140 opened 7 months ago by nicholasjng
  0

• Investigate binding params to benchmarks with weakrefs

  #138 opened 7 months ago by nicholasjng
  0

• Update Artifact Tutorial

  #128 opened 7 months ago by Maciej818
  0

• Hint all `name` arguments as `str` only

  #132 opened 7 months ago by nicholasjng
  1

• Add memo followups to the user guide

  #134 opened 7 months ago by nicholasjng
  0

• Set up best practice for easy record serialization + parametrization + reproducibility

  #105 opened 7 months ago by nicholasjng
  1

• Add global memo cache and eviction API

  #126 opened 7 months ago by nicholasjng
  0

• Add `nnbench.State` object holding current benchmark information, inject into setup/teardown tasks

  #125 opened 7 months ago by nicholasjng
  0

• Parameter representations instead of parameters in benchmark records

  #122 opened 7 months ago by nicholasjng
  3

• Supplying `nnbench.Transform`s directly to record IO

  #131 opened 7 months ago by nicholasjng
  0

• Redesign artifact concept

  #114 opened 8 months ago by nicholasjng
  1

• Enhance API Reference Documentation for nnbench

  #121 opened 8 months ago by leonpawelzik
  1

• Module path construction causes typecheck fails

  #115 opened 8 months ago by nicholasjng
  0

• Improve base `Transform` interface

  #117 opened 8 months ago by nicholasjng
  0

• Make `nnbench.parametrize` work with `functools.partial`

  #96 opened 8 months ago by maxmynter
  0

• Improve ensure unique keys in `Contexts` upon merge.

  #99 opened 8 months ago by maxmynter
  0

• Remove `ArtifactCollection` Wrapper

  #108 opened 8 months ago by maxmynter
  0

• User guide: benchmarking on saved artifacts

  #95 opened 8 months ago by Maciej818
  0

• Implement fsspec artifact loader for nnbench

  #94 opened 8 months ago by Maciej818
  0

• Docs: Versioning considerations

  #106 opened 8 months ago by AdrianoKF
  0

• [umbrella ticket] Add more examples to showcase different applications of the project

  #28 opened 8 months ago by nicholasjng
  1

• Benchmark results do not contain parameters

  #101 opened 8 months ago by nicholasjng
  0

• Add example: Implement benchmarking on saved artifacts

  #63 opened 8 months ago by Maciej818
  1

• Add common ArtifactLoaders to make nnbench provide them out of the box

  #97 opened 8 months ago by maxmynter
  1

• Add user guide: Reporters usage, streaming, creating custom ones, registering

  #37 opened 8 months ago by Maciej818
  1

• Add user guides on various topics to documentation

  #27 opened 9 months ago by nicholasjng
  1

• Streaming results to a database: verify SQL model, decide on what API to go forward

  #86 opened 8 months ago by Maciej818
  2

• Add example: Streaming results to a database

  #66 opened 8 months ago by Maciej818
  2

• Add Streamlit integration Example for `nnbench`

  #45 opened 8 months ago by maxmynter
  0

• Support compression for file reading and writing

  #80 opened 9 months ago by nicholasjng
  0

• Add proper `Context` object, manipulation methods

  #72 opened 9 months ago by nicholasjng
  1

• Add flexible base reporters for file IO, web, and DBs

  #48 opened 9 months ago by nicholasjng
  3

• Add example: Integration with workflow engines

  #67 opened 9 months ago by Maciej818
  0

• Benchmark discovery fails if "./" is not prepended to a relative path

  #69 opened 9 months ago by nicholasjng
  0

• Add some default context values and benchmark metadata

  #56 opened 9 months ago by nicholasjng
  1

• Add test coverage for nnbench core abstractions

  #50 opened 9 months ago by nicholasjng
  0

• Add user guide: Runners

  #38 opened 9 months ago by Maciej818
  0

• Add user guide: Examples of customizing benchmarks and user-defined `nnbench.Parameters` for type safety

  #35 opened 9 months ago by Maciej818
  0

• Finalize names of user-facing abstract classes

  #49 opened 9 months ago by nicholasjng
  0

• Rethink reporter concept

  #24 opened 9 months ago by nicholasjng
  3

• Add user guide: Defining and parametrizing benchmarks by applying decorators

  #34 opened 9 months ago by Maciej818
  2

• Define abstract `BenchmarkReporter` interface

  #47 opened 9 months ago by nicholasjng
  0

• Add `namegen` argument to `@nnbench.{parametrize,product}`

  #40 opened 9 months ago by nicholasjng
  0

• Defining and parametrizing benchmarks by applying decorators (`@benchmark,@parametrize,@product)` - motivation, behavior, and one usage example for each.

  #33 opened 9 months ago by Maciej818
  3

• Add user guide: Best practices on modularizing benchmark code

  #32 opened 9 months ago by Maciej818
  0

• Add user guide: Defining and parametrizing benchmarks by applying decorators

  #31 opened 9 months ago by Maciej818
  1