aws/fmeval

Issues

• [Bug] Radar plot using plotly not displaying

  #316 opened 24 days ago by jcassiojr
  0

• Python 3.11 and 3.12 compatibility?

  #322 opened a month ago by OlivierBinette
  2

• ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-sonnet-20240229" is not supported on this API. Please use the Messages API instead.

  #286 opened 3 months ago by aakash086
  4

• String enums not comparing as expected

  #186 opened 3 months ago by athewsey
  3

• [Feature] Add callback mechanism to evaluation

  #284 opened 4 months ago by acere
  1

• [Feature] Streaming results/progress and summary metrics for faster feedback

  #278 opened 4 months ago by athewsey
  1

• [Feature] JSON export/import for EvalOutput classes

  #268 opened 4 months ago by athewsey
  3

• [Feature] Add system metrics collected during evaluation to eval_output

  #280 opened 4 months ago by acere
  4

• [Feature] Multi-variable prompt templates

  #246 opened 5 months ago by athewsey
  2

• [Feature] Support string output path in `EvalAlgorithmInterface.evaluate(save="...")`

  #165 opened 8 months ago by athewsey
  3

• [Feature] EvalAlgorithmInterface.evaluate should accept a list of DataConfigs for consistency

  #269 opened 4 months ago by athewsey
  1

• [Feature] Image fields for multi-modal models

  #247 opened 5 months ago by athewsey
  1

• Evaluation Algorithm for Recommendations

  #199 opened 6 months ago by mccartni-aws
  4

• Cannot import FactualKnowledge module

  #226 opened 6 months ago by dferguson992
  4

• Add support for system prompt and messages API through ModelRunner.predict()

  #205 opened 6 months ago by gilinachum
  3

• Chinese model and content support

  #202 opened 6 months ago by weinick
  1

• out of memory for summarization accuracy

  #197 opened 7 months ago by gilinachum
  3

• fmeval is incompatible to properly install on (new) SageMaker Studio Python kernel

  #164 opened 7 months ago by athewsey
  2

• Unable to run in Docker container (unable to register worker with raylet)

  #183 opened 7 months ago by athewsey
  4

• [Feature] LLM-based (QA Accuracy) eval algorithm

  #163 opened 8 months ago by athewsey
  2

• Failed running ./devtool: line 14: pre-commit: command not found

  #162 opened 8 months ago by gilinachum
  3

• Dependency issues

  #170 opened 8 months ago by aaronkl
  1