BaseChatModel.agenerate() missing 1 required positional argument: 'messages'

Question

BaseChatModel.agenerate() missing 1 required positional argument: 'messages'

Closed this issue 2 months ago · 0 comments

Checked other resources

This is a bug, not a usage question. For questions, please use the LangChain Forum (https://forum.langchain.com/).
I added a clear and descriptive title that summarizes this issue.
I used the GitHub search to find a similar question and didn't find it.
I am sure that this is a bug in LangChain rather than my code.
The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
I read what a minimal reproducible example is (https://stackoverflow.com/help/minimal-reproducible-example).
I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

Example Code

import pandas as pd
import asyncio
import ast
from ragas.testset import Testset
from ragas import evaluate
from src.eval.RAG import RAG
import os
from dotenv import load_dotenv
from langchain_community.document_loaders import DirectoryLoader
import json
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI
from langchain_openai import OpenAIEmbeddings

load_dotenv()
async def eval_by_llm(
test_set_dir: str,
llm,
):
# ha = pd.read_csv(test_set_dir).to_dict(orient="records")
# os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY")
# # 修正 reference_contexts
# for row in ha:
# if isinstance(row.get("reference_contexts"), str):
# try:
# row["reference_contexts"] = ast.literal_eval(row["reference_contexts"])
# except Exception as e:
# print("轉換失敗:", row["reference_contexts"], e)
# row["reference_contexts"] = []
#
# ts = Testset.from_list(ha).to_list()
# sample_queries = [s["user_input"] for s in ts]
# expected_responses = [e["reference"] for e in ts]
#
# dataset = []
#
# rag = RAG()
#
# #讀文檔
# from src.utils.docs_utils import load_docs
# docs = load_docs()
# rag.load_documents(docs)
#
# for query, reference in zip(sample_queries, expected_responses):
# relevant_docs = rag.get_most_relevant_docs(query)
# response = rag.generate_answer(query, relevant_docs)
# dataset.append(
# {
# "user_input": query,
# "retrieved_contexts": relevant_docs,
# "response": response,
# "reference": reference
# }
# )
#
# with open("datasets.json", "w", encoding="utf-8") as f:
# json.dump(dataset, f, ensure_ascii=False, indent=2)

# 讀檔
with open("tmpds.json", "r", encoding="utf-8") as f:
    dataset = json.load(f)

from ragas import EvaluationDataset
evaluation_dataset = EvaluationDataset.from_list(dataset)

from ragas import evaluate

from ragas.metrics import LLMContextRecall, Faithfulness, FactualCorrectness

result = evaluate(dataset=evaluation_dataset, metrics=[LLMContextRecall(), Faithfulness(), FactualCorrectness()])
print(result)

df = result.to_pandas()
print(df)
df.to_csv(f"tmp.csv", index=False, encoding="utf-8-sig")

Error Message and Stack Trace (if applicable)

/Users/arianx/PycharmProjects/bigproj/.venv/bin/python /Users/arianx/PycharmProjects/bigproj/test__.py 
Evaluating:   0%|          | 0/3 [00:00<?, ?it/s]Exception raised in Job[0]: TypeError(BaseChatModel.agenerate() missing 1 required positional argument: 'messages')
Exception raised in Job[1]: TypeError(BaseChatModel.agenerate() missing 1 required positional argument: 'messages')
Exception raised in Job[2]: TypeError(BaseChatModel.agenerate() missing 1 required positional argument: 'messages')
Evaluating: 100%|██████████| 3/3 [00:00<00:00, 624.87it/s]
{'context_recall': nan, 'faithfulness': nan, 'factual_correctness(mode=f1)': nan}
  user_input retrieved_contexts  ... faithfulness factual_correctness(mode=f1)
0       test             [test]  ...          NaN                          NaN

[1 rows x 7 columns]

Process finished with exit code 0

Description

i stored dataset into json file instead of regenerating it

and this is what dataset look like

[ { "user_input": "test", "retrieved_contexts": ["test" ], "response": "test", "reference": "test" } ]

i have already followed the guide from https://docs.ragas.io/en/stable/getstarted/rag_eval/#collect-evaluation-data so the format of dataset shouldn't be wrong

but it keeps showing me BaseChatModel.agenerate() missing 1 required positional argument: 'messages' and all my outputs are Nan

System Info

/Users/arianx/PycharmProjects/bigproj/.venv/bin/python /Users/arianx/PycharmProjects/bigproj/test__.py

System Information

OS: Darwin
OS Version: Darwin Kernel Version 24.5.0: Tue Apr 22 19:54:43 PDT 2025; root:xnu-11417.121.6~2/RELEASE_ARM64_T8132
Python Version: 3.13.5 (v3.13.5:6cb20a219a8, Jun 11 2025, 12:23:45) [Clang 16.0.0 (clang-1600.0.26.6)]

Package Information

langchain_core: 0.3.75
langchain: 0.3.27
langchain_community: 0.3.29
langsmith: 0.4.15
langchain_huggingface: 0.3.1
langchain_openai: 0.3.30
langchain_text_splitters: 0.3.9

Optional packages not installed

langserve

Other Dependencies

aiohttp<4.0.0,>=3.8.3: Installed. No version info available.
async-timeout<5.0.0,>=4.0.0;: Installed. No version info available.
dataclasses-json<0.7,>=0.6.7: Installed. No version info available.
httpx-sse<1.0.0,>=0.4.0: Installed. No version info available.
httpx<1,>=0.23.0: Installed. No version info available.
huggingface-hub>=0.33.4: Installed. No version info available.
jsonpatch<2.0,>=1.33: Installed. No version info available.
langchain-anthropic;: Installed. No version info available.
langchain-aws;: Installed. No version info available.
langchain-azure-ai;: Installed. No version info available.
langchain-cohere;: Installed. No version info available.
langchain-community;: Installed. No version info available.
langchain-core<1.0.0,>=0.3.70: Installed. No version info available.
langchain-core<1.0.0,>=0.3.72: Installed. No version info available.
langchain-core<1.0.0,>=0.3.74: Installed. No version info available.
langchain-core<2.0.0,>=0.3.75: Installed. No version info available.
langchain-deepseek;: Installed. No version info available.
langchain-fireworks;: Installed. No version info available.
langchain-google-genai;: Installed. No version info available.
langchain-google-vertexai;: Installed. No version info available.
langchain-groq;: Installed. No version info available.
langchain-huggingface;: Installed. No version info available.
langchain-mistralai;: Installed. No version info available.
langchain-ollama;: Installed. No version info available.
langchain-openai;: Installed. No version info available.
langchain-perplexity;: Installed. No version info available.
langchain-text-splitters<1.0.0,>=0.3.9: Installed. No version info available.
langchain-together;: Installed. No version info available.
langchain-xai;: Installed. No version info available.
langchain<2.0.0,>=0.3.27: Installed. No version info available.
langsmith-pyo3>=0.1.0rc2;: Installed. No version info available.
langsmith>=0.1.125: Installed. No version info available.
langsmith>=0.1.17: Installed. No version info available.
langsmith>=0.3.45: Installed. No version info available.
numpy>=1.26.2;: Installed. No version info available.
numpy>=2.1.0;: Installed. No version info available.
openai-agents>=0.0.3;: Installed. No version info available.
openai<2.0.0,>=1.99.9: Installed. No version info available.
opentelemetry-api>=1.30.0;: Installed. No version info available.
opentelemetry-exporter-otlp-proto-http>=1.30.0;: Installed. No version info available.
opentelemetry-sdk>=1.30.0;: Installed. No version info available.
orjson>=3.9.14;: Installed. No version info available.
packaging>=23.2: Installed. No version info available.
pydantic-settings<3.0.0,>=2.10.1: Installed. No version info available.
pydantic<3,>=1: Installed. No version info available.
pydantic<3.0.0,>=2.7.4: Installed. No version info available.
pydantic>=2.7.4: Installed. No version info available.
pytest>=7.0.0;: Installed. No version info available.
PyYAML>=5.3: Installed. No version info available.
requests-toolbelt>=1.0.0: Installed. No version info available.
requests<3,>=2: Installed. No version info available.
requests<3,>=2.32.5: Installed. No version info available.
requests>=2.0.0: Installed. No version info available.
rich>=13.9.4;: Installed. No version info available.
sentence-transformers>=2.6.0;: Installed. No version info available.
SQLAlchemy<3,>=1.4: Installed. No version info available.
tenacity!=8.4.0,<10,>=8.1.0: Installed. No version info available.
tenacity!=8.4.0,<10.0.0,>=8.1.0: Installed. No version info available.
tiktoken<1,>=0.7: Installed. No version info available.
tokenizers>=0.19.1: Installed. No version info available.
transformers>=4.39.0;: Installed. No version info available.
typing-extensions>=4.7: Installed. No version info available.
vcrpy>=7.0.0;: Installed. No version info available.
zstandard>=0.23.0: Installed. No version info available.

Process finished with exit code 0