Cinnamon/kotaemon

[BUG] ValueError while quarying ms graphrag (ValueError: Query vector size 3072 does not match index column size 1536)

Closed this issue · 2 comments

hamzahassan66 commented

Description

I have configured my model settings in .env file. Using Graphrag, indexing works fine (i get create_final_documents🚀 All workflows completed successfully.) log but when I try to query something from my document, I get this error ValueError: Query vector size 3072 does not match index column size 1536

I also tried hard-coding my embedding model to 'text-embedding-3-large' in flowsettings.py and /kotaemon/libs/ktem/ktem/index/file/pipelines.py but didnt work.

How to resolve this issue?

Reproduction steps

conda create -n kotaemon python=3.10
conda activate kotaemon

git clone https://github.com/Cinnamon/kotaemon
cd kotaemon

pip install -e "libs/kotaemon[all]"
pip install -e "libs/ktem"

pip install "graphrag<=0.3.6" future

Screenshots

No response

Logs

Page numbers: 1
Got 1 page thumbnails
Adding documents to doc store
indexing step took 0.09454560279846191
Initializing project at 
/home/hamza_unix/KOTAEMON_new_version/kotaemon/ktem_app_data/user_data/files/graphrag/5557870b-5bcf-49e2-aec2-2e4ae1bd28a0

/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/numpy/core/fromnumeric.py:59: FutureWarning: 'DataFrame.swapaxes' is deprecated and will be removed in a future version. Please use 'DataFrame.transpose' instead.
  return bound(*args, **kwds)
/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/numpy/core/fromnumeric.py:59: FutureWarning: 'DataFrame.swapaxes' is deprecated and will be removed in a future version. Please use 'DataFrame.transpose' instead.
  return bound(*args, **kwds)
/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/numpy/core/fromnumeric.py:59: FutureWarning: 'DataFrame.swapaxes' is deprecated and will be removed in a future version. Please use 'DataFrame.transpose' instead.
  return bound(*args, **kwds)
/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/numpy/core/fromnumeric.py:59: FutureWarning: 'DataFrame.swapaxes' is deprecated and will be removed in a future version. Please use 'DataFrame.transpose' instead.
  return bound(*args, **kwds)
/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/datashaper/engine/verbs/convert.py:72: FutureWarning: errors='ignore' is deprecated and will raise in a future version. Use to_datetime without passing `errors` and catch exceptions explicitly instead
  datetime_column = pd.to_datetime(column, errors="ignore")
/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/datashaper/engine/verbs/convert.py:72: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
  datetime_column = pd.to_datetime(column, errors="ignore")
/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/datashaper/engine/verbs/convert.py:72: FutureWarning: errors='ignore' is deprecated and will raise in a future version. Use to_datetime without passing `errors` and catch exceptions explicitly instead
  datetime_column = pd.to_datetime(column, errors="ignore")
User-id: 1, can see public conversations: True
Session reasoning type None use mindmap (default) use citation (default) language (default)
Session LLM 
Reasoning class <class 'ktem.reasoning.simple.FullQAPipeline'>
Reasoning state {'app': {'regen': False}, 'pipeline': {}}
Thinking ...
Retrievers [DocumentRetrievalPipeline(DS=<kotaemon.storages.docstores.lancedb.LanceDBDocumentStore object at 0x7f7c01093010>, FSPath=PosixPath('/home/hamza_unix/KOTAEMON_new_version/kotaemon/ktem_app_data/user_data/files/index_1'), Index=<class 'ktem.index.file.index.IndexTable'>, Source=<class 'ktem.index.file.index.Source'>, VS=<kotaemon.storages.vectorstores.chroma.ChromaVectorStore object at 0x7f7c01092e00>, get_extra_table=False, llm_scorer=LLMTrulensScoring(concurrent=True, normalize=10, prompt_template=<kotaemon.llms.prompts.template.PromptTemplate object at 0x7f7b640fcb80>, system_prompt_template=<kotaemon.llms.prompts.template.PromptTemplate object at 0x7f7b640fccd0>, top_k=3, user_prompt_template=<kotaemon.llms.prompts.template.PromptTemplate object at 0x7f7b640fc400>), mmr=False, rerankers=[CohereReranking(cohere_api_key='eXNwNuyjzqeVampQ8W5IumFtvay3yPz6j2aXh6hV', model_name='rerank-multilingual-v2.0')], retrieval_mode='hybrid', top_k=10, user_id=1), GraphRAGRetrieverPipeline(DS=<theflow.base.unset_ object at 0x7f7c675720e0>, FSPath=<theflow.base.unset_ object at 0x7f7c675720e0>, Index=<class 'ktem.index.file.index.IndexTable'>, Source=<theflow.base.unset_ object at 0x7f7c675720e0>, VS=<theflow.base.unset_ object at 0x7f7c675720e0>, file_ids=['86ca9c2c-0e4b-4e1e-9bd9-8cdfc07ebef1'], user_id=<theflow.base.unset_ object at 0x7f7c675720e0>)]
searching in doc_ids []
[2024-11-30T13:30:39Z WARN  lance::dataset] No existing dataset at /home/hamza_unix/KOTAEMON_new_version/kotaemon/ktem_app_data/user_data/files/graphrag/5557870b-5bcf-49e2-aec2-2e4ae1bd28a0/output/lancedb/entity_description_embeddings.lance, it will be created
Entity count: 4
Traceback (most recent call last):
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/gradio/queueing.py", line 575, in process_events
    response = await route_utils.call_process_api(
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/gradio/route_utils.py", line 276, in call_process_api
    output = await app.get_blocks().process_api(
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/gradio/blocks.py", line 1923, in process_api
    result = await self.call_function(
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/gradio/blocks.py", line 1520, in call_function
    prediction = await utils.async_iteration(iterator)
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/gradio/utils.py", line 663, in async_iteration
    return await iterator.__anext__()
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/gradio/utils.py", line 656, in __anext__
    return await anyio.to_thread.run_sync(
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2441, in run_sync_in_worker_thread
    return await future
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 943, in run
    result = context.run(func, *args)
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/gradio/utils.py", line 639, in run_sync_iterator_async
    return next(iterator)
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/gradio/utils.py", line 801, in gen_wrapper
    response = next(iterator)
  File "/home/hamza_unix/KOTAEMON_new_version/kotaemon/libs/ktem/ktem/pages/chat/__init__.py", line 974, in chat_fn
    for response in pipeline.stream(chat_input, conversation_id, chat_history):
  File "/home/hamza_unix/KOTAEMON_new_version/kotaemon/libs/ktem/ktem/reasoning/simple.py", line 287, in stream
    docs, infos = self.retrieve(message, history)
  File "/home/hamza_unix/KOTAEMON_new_version/kotaemon/libs/ktem/ktem/reasoning/simple.py", line 130, in retrieve
    retriever_docs = retriever_node(text=query)
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/theflow/base.py", line 1097, in __call__
    raise e from None
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/theflow/base.py", line 1088, in __call__
    output = self.fl.exec(func, args, kwargs)
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/theflow/backends/base.py", line 151, in exec
    return run(*args, **kwargs)
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/theflow/middleware.py", line 144, in __call__
    raise e from None
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/theflow/middleware.py", line 141, in __call__
    _output = self.next_call(*args, **kwargs)
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/theflow/middleware.py", line 117, in __call__
    return self.next_call(*args, **kwargs)
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/theflow/base.py", line 1017, in _runx
    return self.run(*args, **kwargs)
  File "/home/hamza_unix/KOTAEMON_new_version/kotaemon/libs/ktem/ktem/index/file/graph/pipelines.py", line 370, in run
    context_text, context_records = context_builder.build_context(
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/graphrag/query/structured_search/local_search/mixed_context.py", line 140, in build_context
    selected_entities = map_query_to_entities(
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/graphrag/query/context_builder/entity_extraction.py", line 55, in map_query_to_entities
    search_results = text_embedding_vectorstore.similarity_search_by_text(
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/graphrag/vector_stores/lancedb.py", line 120, in similarity_search_by_text
    return self.similarity_search_by_vector(query_embedding, k)
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/graphrag/vector_stores/lancedb.py", line 99, in similarity_search_by_vector
    .to_list()
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/lancedb/query.py", line 320, in to_list
    return self.to_arrow().to_pylist()
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/lancedb/query.py", line 647, in to_arrow
    return self.to_batches().read_all()
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/lancedb/query.py", line 678, in to_batches
    result_set = self._table._execute_query(query, batch_size)
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/lancedb/table.py", line 1742, in _execute_query
    return ds.scanner(
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/lance/dataset.py", line 369, in scanner
    builder = builder.nearest(**nearest)
  File "/home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages/lance/dataset.py", line 2453, in nearest
    raise ValueError(
ValueError: Query vector size 3072 does not match index column size 1536
User-id: 1, can see public conversations: True

Browsers

Chrome

OS

Linux

Additional information

While installing graphrag, I get these logs:

(ktem) hamza_unix@Asus-Zenbook-Security-Tagged:/KOTAEMON_new_version/kotaemon$ pip install "graphrag<=0.3.6" future
Collecting graphrag<=0.3.6
Using cached graphrag-0.3.6-py3-none-any.whl.metadata (6.0 kB)
Collecting future
Using cached future-1.0.0-py3-none-any.whl.metadata (4.0 kB)
Collecting aiofiles<25.0.0,>=24.1.0 (from graphrag<=0.3.6)
Using cached aiofiles-24.1.0-py3-none-any.whl.metadata (10 kB)
Collecting aiolimiter<2.0.0,>=1.1.0 (from graphrag<=0.3.6)
Using cached aiolimiter-1.1.0-py3-none-any.whl.metadata (4.5 kB)
Collecting azure-identity<2.0.0,>=1.17.1 (from graphrag<=0.3.6)
Using cached azure_identity-1.19.0-py3-none-any.whl.metadata (80 kB)
Collecting azure-search-documents<12.0.0,>=11.4.0 (from graphrag<=0.3.6)
Using cached azure_search_documents-11.5.2-py3-none-any.whl.metadata (23 kB)
Collecting azure-storage-blob<13.0.0,>=12.22.0 (from graphrag<=0.3.6)
Using cached azure_storage_blob-12.24.0-py3-none-any.whl.metadata (26 kB)
Collecting datashaper<0.0.50,>=0.0.49 (from graphrag<=0.3.6)
Using cached datashaper-0.0.49-py3-none-any.whl.metadata (3.7 kB)
Collecting devtools<0.13.0,>=0.12.2 (from graphrag<=0.3.6)
Using cached devtools-0.12.2-py3-none-any.whl.metadata (4.8 kB)
Collecting environs<12.0.0,>=11.0.0 (from graphrag<=0.3.6)
Using cached environs-11.2.1-py3-none-any.whl.metadata (13 kB)
Collecting graspologic<4.0.0,>=3.4.1 (from graphrag<=0.3.6)
Using cached graspologic-3.4.1-py3-none-any.whl.metadata (5.8 kB)
Collecting json-repair<0.29.0,>=0.28.4 (from graphrag<=0.3.6)
Using cached json_repair-0.28.4-py3-none-any.whl.metadata (9.0 kB)
Collecting lancedb<0.14.0,>=0.13.0 (from graphrag<=0.3.6)
Using cached lancedb-0.13.0-cp38-abi3-manylinux_2_28_x86_64.whl.metadata (4.8 kB)
Requirement already satisfied: matplotlib<4.0.0,>=3.9.0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from graphrag<=0.3.6) (3.9.3)
Requirement already satisfied: networkx<4,>=3 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from graphrag<=0.3.6) (3.4.2)
Requirement already satisfied: nltk==3.9.1 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from graphrag<=0.3.6) (3.9.1)
Requirement already satisfied: numpy<2.0.0,>=1.25.2 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from graphrag<=0.3.6) (1.26.4)
Requirement already satisfied: openai<2.0.0,>=1.46.1 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from graphrag<=0.3.6) (1.55.3)
Requirement already satisfied: pandas<3.0.0,>=2.2.2 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from graphrag<=0.3.6) (2.2.3)
Collecting pyaml-env<2.0.0,>=1.2.1 (from graphrag<=0.3.6)
Using cached pyaml_env-1.2.1-py3-none-any.whl.metadata (12 kB)
Collecting pyarrow<16.0.0,>=15.0.0 (from graphrag<=0.3.6)
Using cached pyarrow-15.0.2-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (3.0 kB)
Requirement already satisfied: pydantic<3.0.0,>=2.9.2 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from graphrag<=0.3.6) (2.9.2)
Requirement already satisfied: python-dotenv<2.0.0,>=1.0.0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from graphrag<=0.3.6) (1.0.1)
Requirement already satisfied: pyyaml<7.0.0,>=6.0.2 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from graphrag<=0.3.6) (6.0.2)
Requirement already satisfied: rich<14.0.0,>=13.6.0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from graphrag<=0.3.6) (13.9.4)
Collecting tenacity<10.0.0,>=9.0.0 (from graphrag<=0.3.6)
Using cached tenacity-9.0.0-py3-none-any.whl.metadata (1.2 kB)
Collecting tiktoken<0.8.0,>=0.7.0 (from graphrag<=0.3.6)
Using cached tiktoken-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Requirement already satisfied: typing-extensions<5.0.0,>=4.12.2 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from graphrag<=0.3.6) (4.12.2)
Collecting umap-learn<0.6.0,>=0.5.6 (from graphrag<=0.3.6)
Using cached umap_learn-0.5.7-py3-none-any.whl.metadata (21 kB)
Requirement already satisfied: click in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from nltk==3.9.1->graphrag<=0.3.6) (8.1.7)
Requirement already satisfied: joblib in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from nltk==3.9.1->graphrag<=0.3.6) (1.4.2)
Requirement already satisfied: regex>=2021.8.3 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from nltk==3.9.1->graphrag<=0.3.6) (2024.11.6)
Requirement already satisfied: tqdm in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from nltk==3.9.1->graphrag<=0.3.6) (4.67.1)
Requirement already satisfied: azure-core>=1.31.0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from azure-identity<2.0.0,>=1.17.1->graphrag<=0.3.6) (1.32.0)
Requirement already satisfied: cryptography>=2.5 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from azure-identity<2.0.0,>=1.17.1->graphrag<=0.3.6) (44.0.0)
Collecting msal>=1.30.0 (from azure-identity<2.0.0,>=1.17.1->graphrag<=0.3.6)
Using cached msal-1.31.1-py3-none-any.whl.metadata (11 kB)
Collecting msal-extensions>=1.2.0 (from azure-identity<2.0.0,>=1.17.1->graphrag<=0.3.6)
Using cached msal_extensions-1.2.0-py3-none-any.whl.metadata (7.6 kB)
Collecting azure-common>=1.1 (from azure-search-documents<12.0.0,>=11.4.0->graphrag<=0.3.6)
Using cached azure_common-1.1.28-py2.py3-none-any.whl.metadata (5.0 kB)
Requirement already satisfied: isodate>=0.6.0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from azure-search-documents<12.0.0,>=11.4.0->graphrag<=0.3.6) (0.7.2)
Requirement already satisfied: diskcache<6.0.0,>=5.6.3 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from datashaper<0.0.50,>=0.0.49->graphrag<=0.3.6) (5.6.3)
Collecting jsonschema<5.0.0,>=4.21.1 (from datashaper<0.0.50,>=0.0.49->graphrag<=0.3.6)
Using cached jsonschema-4.23.0-py3-none-any.whl.metadata (7.9 kB)
Collecting asttokens<3.0.0,>=2.0.0 (from devtools<0.13.0,>=0.12.2->graphrag<=0.3.6)
Using cached asttokens-2.4.1-py2.py3-none-any.whl.metadata (5.2 kB)
Requirement already satisfied: executing>=1.1.1 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from devtools<0.13.0,>=0.12.2->graphrag<=0.3.6) (2.1.0)
Requirement already satisfied: pygments>=2.15.0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from devtools<0.13.0,>=0.12.2->graphrag<=0.3.6) (2.18.0)
Requirement already satisfied: marshmallow>=3.13.0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from environs<12.0.0,>=11.0.0->graphrag<=0.3.6) (3.23.1)
Collecting POT<0.10,>=0.9 (from graspologic<4.0.0,>=3.4.1->graphrag<=0.3.6)
Using cached POT-0.9.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (34 kB)
Collecting anytree<3.0.0,>=2.12.1 (from graspologic<4.0.0,>=3.4.1->graphrag<=0.3.6)
Using cached anytree-2.12.1-py3-none-any.whl.metadata (8.1 kB)
Collecting beartype<0.19.0,>=0.18.5 (from graspologic<4.0.0,>=3.4.1->graphrag<=0.3.6)
Using cached beartype-0.18.5-py3-none-any.whl.metadata (30 kB)
Collecting gensim<5.0.0,>=4.3.2 (from graspologic<4.0.0,>=3.4.1->graphrag<=0.3.6)
Using cached gensim-4.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.2 kB)
Collecting graspologic-native<2.0.0,>=1.2.1 (from graspologic<4.0.0,>=3.4.1->graphrag<=0.3.6)
Using cached graspologic_native-1.2.1-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.9 kB)
Collecting hyppo<0.5.0,>=0.4.0 (from graspologic<4.0.0,>=3.4.1->graphrag<=0.3.6)
Using cached hyppo-0.4.0-py3-none-any.whl.metadata (1.7 kB)
Requirement already satisfied: scikit-learn<2.0.0,>=1.4.2 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from graspologic<4.0.0,>=3.4.1->graphrag<=0.3.6) (1.5.2)
Collecting scipy==1.12.0 (from graspologic<4.0.0,>=3.4.1->graphrag<=0.3.6)
Using cached scipy-1.12.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
Collecting seaborn<0.14.0,>=0.13.2 (from graspologic<4.0.0,>=3.4.1->graphrag<=0.3.6)
Using cached seaborn-0.13.2-py3-none-any.whl.metadata (5.4 kB)
Collecting statsmodels<0.15.0,>=0.14.2 (from graspologic<4.0.0,>=3.4.1->graphrag<=0.3.6)
Using cached statsmodels-0.14.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.2 kB)
Requirement already satisfied: deprecation in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from lancedb<0.14.0,>=0.13.0->graphrag<=0.3.6) (2.1.0)
Collecting pylance==0.17.0 (from lancedb<0.14.0,>=0.13.0->graphrag<=0.3.6)
Using cached pylance-0.17.0-cp39-abi3-manylinux_2_28_x86_64.whl.metadata (7.2 kB)
Requirement already satisfied: requests>=2.31.0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from lancedb<0.14.0,>=0.13.0->graphrag<=0.3.6) (2.32.3)
Collecting retry>=0.9.2 (from lancedb<0.14.0,>=0.13.0->graphrag<=0.3.6)
Using cached retry-0.9.2-py2.py3-none-any.whl.metadata (5.8 kB)
Requirement already satisfied: attrs>=21.3.0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from lancedb<0.14.0,>=0.13.0->graphrag<=0.3.6) (24.2.0)
Requirement already satisfied: packaging in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from lancedb<0.14.0,>=0.13.0->graphrag<=0.3.6) (24.2)
Requirement already satisfied: cachetools in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from lancedb<0.14.0,>=0.13.0->graphrag<=0.3.6) (5.5.0)
Requirement already satisfied: overrides>=0.7 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from lancedb<0.14.0,>=0.13.0->graphrag<=0.3.6) (7.7.0)
Requirement already satisfied: contourpy>=1.0.1 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from matplotlib<4.0.0,>=3.9.0->graphrag<=0.3.6) (1.3.1)
Requirement already satisfied: cycler>=0.10 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from matplotlib<4.0.0,>=3.9.0->graphrag<=0.3.6) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from matplotlib<4.0.0,>=3.9.0->graphrag<=0.3.6) (4.55.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from matplotlib<4.0.0,>=3.9.0->graphrag<=0.3.6) (1.4.7)
Requirement already satisfied: pillow>=8 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from matplotlib<4.0.0,>=3.9.0->graphrag<=0.3.6) (10.4.0)
Requirement already satisfied: pyparsing>=2.3.1 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from matplotlib<4.0.0,>=3.9.0->graphrag<=0.3.6) (3.2.0)
Requirement already satisfied: python-dateutil>=2.7 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from matplotlib<4.0.0,>=3.9.0->graphrag<=0.3.6) (2.8.2)
Requirement already satisfied: anyio<5,>=3.5.0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from openai<2.0.0,>=1.46.1->graphrag<=0.3.6) (4.6.2.post1)
Requirement already satisfied: distro<2,>=1.7.0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from openai<2.0.0,>=1.46.1->graphrag<=0.3.6) (1.9.0)
Requirement already satisfied: httpx<1,>=0.23.0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from openai<2.0.0,>=1.46.1->graphrag<=0.3.6) (0.28.0)
Requirement already satisfied: jiter<1,>=0.4.0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from openai<2.0.0,>=1.46.1->graphrag<=0.3.6) (0.8.0)
Requirement already satisfied: sniffio in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from openai<2.0.0,>=1.46.1->graphrag<=0.3.6) (1.3.1)
Requirement already satisfied: pytz>=2020.1 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from pandas<3.0.0,>=2.2.2->graphrag<=0.3.6) (2024.2)
Requirement already satisfied: tzdata>=2022.7 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from pandas<3.0.0,>=2.2.2->graphrag<=0.3.6) (2024.2)
Requirement already satisfied: annotated-types>=0.6.0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from pydantic<3.0.0,>=2.9.2->graphrag<=0.3.6) (0.7.0)
Requirement already satisfied: pydantic-core==2.23.4 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from pydantic<3.0.0,>=2.9.2->graphrag<=0.3.6) (2.23.4)
Requirement already satisfied: markdown-it-py>=2.2.0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from rich<14.0.0,>=13.6.0->graphrag<=0.3.6) (3.0.0)
Requirement already satisfied: numba>=0.51.2 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from umap-learn<0.6.0,>=0.5.6->graphrag<=0.3.6) (0.60.0)
Requirement already satisfied: pynndescent>=0.5 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from umap-learn<0.6.0,>=0.5.6->graphrag<=0.3.6) (0.5.13)
Requirement already satisfied: idna>=2.8 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from anyio<5,>=3.5.0->openai<2.0.0,>=1.46.1->graphrag<=0.3.6) (3.10)
Requirement already satisfied: exceptiongroup>=1.0.2 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from anyio<5,>=3.5.0->openai<2.0.0,>=1.46.1->graphrag<=0.3.6) (1.2.2)
Requirement already satisfied: six in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from anytree<3.0.0,>=2.12.1->graspologic<4.0.0,>=3.4.1->graphrag<=0.3.6) (1.16.0)
Requirement already satisfied: cffi>=1.12 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from cryptography>=2.5->azure-identity<2.0.0,>=1.17.1->graphrag<=0.3.6) (1.17.1)
Collecting smart-open>=1.8.1 (from gensim<5.0.0,>=4.3.2->graspologic<4.0.0,>=3.4.1->graphrag<=0.3.6)
Using cached smart_open-7.0.5-py3-none-any.whl.metadata (24 kB)
Requirement already satisfied: certifi in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from httpx<1,>=0.23.0->openai<2.0.0,>=1.46.1->graphrag<=0.3.6) (2024.8.30)
Requirement already satisfied: httpcore==1.* in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from httpx<1,>=0.23.0->openai<2.0.0,>=1.46.1->graphrag<=0.3.6) (1.0.7)
Requirement already satisfied: h11<0.15,>=0.13 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai<2.0.0,>=1.46.1->graphrag<=0.3.6) (0.14.0)
Collecting autograd>=1.3 (from hyppo<0.5.0,>=0.4.0->graspologic<4.0.0,>=3.4.1->graphrag<=0.3.6)
Using cached autograd-1.7.0-py3-none-any.whl.metadata (7.5 kB)
Collecting jsonschema-specifications>=2023.03.6 (from jsonschema<5.0.0,>=4.21.1->datashaper<0.0.50,>=0.0.49->graphrag<=0.3.6)
Using cached jsonschema_specifications-2024.10.1-py3-none-any.whl.metadata (3.0 kB)
Collecting referencing>=0.28.4 (from jsonschema<5.0.0,>=4.21.1->datashaper<0.0.50,>=0.0.49->graphrag<=0.3.6)
Using cached referencing-0.35.1-py3-none-any.whl.metadata (2.8 kB)
Collecting rpds-py>=0.7.1 (from jsonschema<5.0.0,>=4.21.1->datashaper<0.0.50,>=0.0.49->graphrag<=0.3.6)
Using cached rpds_py-0.21.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)
Requirement already satisfied: mdurl=0.1 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich<14.0.0,>=13.6.0->graphrag<=0.3.6) (0.1.2)
Collecting PyJWT<3,>=1.0.0 (from PyJWT[crypto]<3,>=1.0.0->msal>=1.30.0->azure-identity<2.0.0,>=1.17.1->graphrag<=0.3.6)
Using cached PyJWT-2.10.1-py3-none-any.whl.metadata (4.0 kB)
Requirement already satisfied: portalocker<3,>=1.4 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from msal-extensions>=1.2.0->azure-identity<2.0.0,>=1.17.1->graphrag<=0.3.6) (2.10.1)
Requirement already satisfied: llvmlite<0.44,>=0.43.0dev0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from numba>=0.51.2->umap-learn<0.6.0,>=0.5.6->graphrag<=0.3.6) (0.43.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from requests>=2.31.0->lancedb<0.14.0,>=0.13.0->graphrag<=0.3.6) (3.4.0)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from requests>=2.31.0->lancedb<0.14.0,>=0.13.0->graphrag<=0.3.6) (2.2.3)
Requirement already satisfied: decorator>=3.4.2 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from retry>=0.9.2->lancedb<0.14.0,>=0.13.0->graphrag<=0.3.6) (5.1.1)
Collecting py<2.0.0,>=1.4.26 (from retry>=0.9.2->lancedb<0.14.0,>=0.13.0->graphrag<=0.3.6)
Using cached py-1.11.0-py2.py3-none-any.whl.metadata (2.8 kB)
Requirement already satisfied: threadpoolctl>=3.1.0 in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from scikit-learn<2.0.0,>=1.4.2->graspologic<4.0.0,>=3.4.1->graphrag<=0.3.6) (3.5.0)
Collecting patsy>=0.5.6 (from statsmodels<0.15.0,>=0.14.2->graspologic<4.0.0,>=3.4.1->graphrag<=0.3.6)
Using cached patsy-1.0.1-py2.py3-none-any.whl.metadata (3.3 kB)
Requirement already satisfied: pycparser in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=2.5->azure-identity<2.0.0,>=1.17.1->graphrag<=0.3.6) (2.22)
Requirement already satisfied: wrapt in /home/hamza_unix/miniconda3/envs/ktem/lib/python3.10/site-packages (from smart-open>=1.8.1->gensim<5.0.0,>=4.3.2->graspologic<4.0.0,>=3.4.1->graphrag<=0.3.6) (1.17.0)
Using cached graphrag-0.3.6-py3-none-any.whl (389 kB)
Using cached future-1.0.0-py3-none-any.whl (491 kB)
Using cached aiofiles-24.1.0-py3-none-any.whl (15 kB)
Using cached aiolimiter-1.1.0-py3-none-any.whl (7.2 kB)
Using cached azure_identity-1.19.0-py3-none-any.whl (187 kB)
Using cached azure_search_documents-11.5.2-py3-none-any.whl (298 kB)
Using cached azure_storage_blob-12.24.0-py3-none-any.whl (408 kB)
Using cached datashaper-0.0.49-py3-none-any.whl (71 kB)
Using cached devtools-0.12.2-py3-none-any.whl (19 kB)
Using cached environs-11.2.1-py3-none-any.whl (12 kB)
Using cached graspologic-3.4.1-py3-none-any.whl (5.2 MB)
Using cached scipy-1.12.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (38.4 MB)
Using cached json_repair-0.28.4-py3-none-any.whl (13 kB)
Using cached lancedb-0.13.0-cp38-abi3-manylinux_2_28_x86_64.whl (24.2 MB)
Using cached pylance-0.17.0-cp39-abi3-manylinux_2_28_x86_64.whl (29.2 MB)
Using cached pyaml_env-1.2.1-py3-none-any.whl (9.0 kB)
Using cached pyarrow-15.0.2-cp310-cp310-manylinux_2_28_x86_64.whl (38.3 MB)
Using cached tenacity-9.0.0-py3-none-any.whl (28 kB)
Using cached tiktoken-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Using cached umap_learn-0.5.7-py3-none-any.whl (88 kB)
Using cached anytree-2.12.1-py3-none-any.whl (44 kB)
Using cached asttokens-2.4.1-py2.py3-none-any.whl (27 kB)
Using cached azure_common-1.1.28-py2.py3-none-any.whl (14 kB)
Using cached beartype-0.18.5-py3-none-any.whl (917 kB)
Using cached gensim-4.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26.5 MB)
Using cached graspologic_native-1.2.1-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
Using cached hyppo-0.4.0-py3-none-any.whl (146 kB)
Using cached jsonschema-4.23.0-py3-none-any.whl (88 kB)
Using cached msal-1.31.1-py3-none-any.whl (113 kB)
Using cached msal_extensions-1.2.0-py3-none-any.whl (19 kB)
Using cached POT-0.9.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (865 kB)
Using cached retry-0.9.2-py2.py3-none-any.whl (8.0 kB)
Using cached seaborn-0.13.2-py3-none-any.whl (294 kB)
Using cached statsmodels-0.14.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.8 MB)
Using cached autograd-1.7.0-py3-none-any.whl (52 kB)
Using cached jsonschema_specifications-2024.10.1-py3-none-any.whl (18 kB)
Using cached patsy-1.0.1-py2.py3-none-any.whl (232 kB)
Using cached py-1.11.0-py2.py3-none-any.whl (98 kB)
Using cached PyJWT-2.10.1-py3-none-any.whl (22 kB)
Using cached referencing-0.35.1-py3-none-any.whl (26 kB)
Using cached rpds_py-0.21.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (360 kB)
Using cached smart_open-7.0.5-py3-none-any.whl (61 kB)
Installing collected packages: azure-common, tenacity, smart-open, scipy, rpds-py, PyJWT, pyarrow, pyaml-env, py, patsy, json-repair, graspologic-native, future, beartype, autograd, asttokens, anytree, aiolimiter, aiofiles, tiktoken, retry, referencing, pylance, POT, gensim, environs, devtools, statsmodels, seaborn, lancedb, jsonschema-specifications, hyppo, azure-storage-blob, azure-search-documents, umap-learn, msal, jsonschema, msal-extensions, graspologic, datashaper, azure-identity, graphrag
Attempting uninstall: tenacity
Found existing installation: tenacity 8.2.3
Uninstalling tenacity-8.2.3:
Successfully uninstalled tenacity-8.2.3
Attempting uninstall: scipy
Found existing installation: scipy 1.14.1
Uninstalling scipy-1.14.1:
Successfully uninstalled scipy-1.14.1
Attempting uninstall: pyarrow
Found existing installation: pyarrow 18.1.0
Uninstalling pyarrow-18.1.0:
Successfully uninstalled pyarrow-18.1.0
Attempting uninstall: asttokens
Found existing installation: asttokens 3.0.0
Uninstalling asttokens-3.0.0:
Successfully uninstalled asttokens-3.0.0
Attempting uninstall: aiofiles
Found existing installation: aiofiles 23.2.1
Uninstalling aiofiles-23.2.1:
Successfully uninstalled aiofiles-23.2.1
Attempting uninstall: tiktoken
Found existing installation: tiktoken 0.8.0
Uninstalling tiktoken-0.8.0:
Successfully uninstalled tiktoken-0.8.0
Attempting uninstall: pylance
Found existing installation: pylance 0.19.2
Uninstalling pylance-0.19.2:
Successfully uninstalled pylance-0.19.2
Attempting uninstall: lancedb
Found existing installation: lancedb 0.16.0
Uninstalling lancedb-0.16.0:
Successfully uninstalled lancedb-0.16.0
Attempting uninstall: umap-learn
Found existing installation: umap-learn 0.5.5
Uninstalling umap-learn-0.5.5:
Successfully uninstalled umap-learn-0.5.5
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
gradio 4.39.0 requires aiofiles<24.0,>=22.0, but you have aiofiles 24.1.0 which is incompatible.
kotaemon 0.9.2 requires tenacity<8.3,>=8.2.3, but you have tenacity 9.0.0 which is incompatible.
kotaemon 0.9.2 requires umap-learn==0.5.5, but you have umap-learn 0.5.7 which is incompatible.
langchain 0.2.15 requires tenacity!=8.4.0,<9.0.0,>=8.1.0, but you have tenacity 9.0.0 which is incompatible.
langchain-community 0.2.11 requires tenacity!=8.4.0,<9.0.0,>=8.1.0, but you have tenacity 9.0.0 which is incompatible.
langchain-core 0.2.43 requires tenacity!=8.4.0,<9.0.0,>=8.1.0, but you have tenacity 9.0.0 which is incompatible.
llama-index-core 0.10.68.post1 requires tenacity!=8.4.0,<9.0.0,>=8.2.0, but you have tenacity 9.0.0 which is incompatible.
llama-index-legacy 0.9.48.post4 requires tenacity<9.0.0,>=8.2.0, but you have tenacity 9.0.0 which is incompatible.
Successfully installed POT-0.9.5 PyJWT-2.10.1 aiofiles-24.1.0 aiolimiter-1.1.0 anytree-2.12.1 asttokens-2.4.1 autograd-1.7.0 azure-common-1.1.28 azure-identity-1.19.0 azure-search-documents-11.5.2 azure-storage-blob-12.24.0 beartype-0.18.5 datashaper-0.0.49 devtools-0.12.2 environs-11.2.1 future-1.0.0 gensim-4.3.3 graphrag-0.3.6 graspologic-3.4.1 graspologic-native-1.2.1 hyppo-0.4.0 json-repair-0.28.4 jsonschema-4.23.0 jsonschema-specifications-2024.10.1 lancedb-0.13.0 msal-1.31.1 msal-extensions-1.2.0 patsy-1.0.1 py-1.11.0 pyaml-env-1.2.1 pyarrow-15.0.2 pylance-0.17.0 referencing-0.35.1 retry-0.9.2 rpds-py-0.21.0 scipy-1.12.0 seaborn-0.13.2 smart-open-7.0.5 statsmodels-0.14.4 tenacity-9.0.0 tiktoken-0.7.0 umap-learn-0.5.7

taprosoft commented

Make sure that GRAPHRAG_EMBEDDING_MODEL="text-embedding-3-small" env var is set to make the embedding model consistent between indexing and retrieval.
If you want to use other model, use USE_CUSTOMIZED_GRAPHRAG_SETTING=true and edit GraphRAG configuration file settings.yaml.example. https://microsoft.github.io/graphrag/config/yaml/

hamzahassan66 commented

Resolved. Thank you