llm-as-a-judge
There are 27 repositories under llm-as-a-judge topic.
Agenta-AI/agenta
The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place.
prometheus-eval/prometheus-eval
Evaluate your LLM's response with Prometheus and GPT4 💯
metauto-ai/agent-as-a-judge
⚖️ The First Coding Agent-as-a-Judge
IAAR-Shanghai/xFinder
[ICLR 2025] xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation
IAAR-Shanghai/xVerify
xVerify: Efficient Answer Verifier for Reasoning Model Evaluations
martin-wey/CodeUltraFeedback
CodeUltraFeedback: aligning large language models to coding preferences (TOSEM 2025)
KID-22/LLM-IR-Bias-Fairness-Survey
This is the repo for the survey of Bias and Fairness in IR with LLMs.
MJ-Bench/MJ-Bench
Official implementation for "MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?"
docling-project/docling-sdg
A set of tools to create synthetically-generated data from documents
zhaochen0110/Timo
Code and data for "Timo: Towards Better Temporal Reasoning for Language Models" (COLM 2024)
minnesotanlp/cobbler
Code and data for Koo et al's ACL 2024 paper "Benchmarking Cognitive Biases in Large Language Models as Evaluators"
PKU-ONELab/Themis
The official repository for our EMNLP 2024 paper, Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability.
OussamaSghaier/CuREV
Harnessing Large Language Models for Curated Code Reviews
root-signals/rs-python-sdk
Root Signals Python SDK
root-signals/root-signals-mcp
MCP for Root Signals Evaluation Platform
aws-samples/genai-system-evaluation
A set of examples demonstrating how to evaluate Generative AI augmented systems using traditional information retrieval and LLM-As-A-Judge validation techniques
HillPhelmuth/LlmAsJudgeEvalPlugins
LLM-as-judge evals as Semantic Kernel Plugins
UMass-Meta-LLM-Eval/llm_eval
A comprehensive study of the LLM-as-a-judge paradigm in a controlled setup that reveals new results about its strengths and weaknesses.
PKU-ONELab/LLM-evaluator-reliability
The official repository for our ACL 2024 paper: Are LLM-based Evaluators Confusing NLG Quality Criteria?
emory-irlab/conqret-rag
Controversial Questions for Argumentation and Retrieval
aws-samples/model-as-a-judge-eval
Notebooks for evaluating LLM based applications using the Model (LLM) as a judge pattern.
romaingrx/llm-as-a-jailbreak-judge
Explore techniques to use small models as jailbreaking judges
djokester/groqeval
Use groq for evaluations
herrxy/LLM-evaluator-reliability
The official repository for our ACL 2024 paper: Are LLM-based Evaluators Confusing NLG Quality Criteria?
catarinaopires/mllm-as-a-judge
ImageCLEFmed 2013 case-based retrieval task relevance judgments expansion via an MLLM-as-a-Judge approach.
cortex359/matse-aachen.seminararbeit
Code zur Bachelor-Seminararbeit: Paarweises Ranking für automatische Textbewertung mit LLMs
rafaelsandroni/antibodies
Antibodies for LLMs hallucinations (grouping LLM as a judge, NLI, reward models)