Query execution hangs for 07_custom.py
Opened this issue · 3 comments
Hi there, I was trying to get the 07_custom.py program to run with the facebook/opt-iml-1.3b model and I can see it loads the cache correctly and I put in enough print statements to see that it also was able to get the LLMPredictor, create the service context, and load the index from disk. However when it tries to call execute_query the program seemingly hangs. I can see my RAM usage spike for an extended period of time but no matter how long I wait (20 minutes?) I don't get a response from the model. Note that I am running with an AMD GPU so when creating the pipeline I removed the CUDA device specification because as far as I can tell CUDA Is not supported with AMD GPUs. Do I need a more powerful computer or CUDA to run this?
Here are my specifications:
OS: Windows 11
Processor AMD Ryzen 7 5800H with Radeon Graphics 3.20 GHz
Installed RAM 16.0 GB (13.9 GB usable)
Device ID XXXXXXXXXXXXXX
Product ID 00342-20715-34612-AAOEM
System type 64-bit operating system, x64-based processor
GPU 0: AMD radeon RX 6600M
GPU1: AMD Radeon(TM) Graphcis
Pen and touch Pen support
Thanks for your help!
I'm having similar issues but on MacBook pro M1. It took 2808 seconds to rerun query results. Here is output:
Loading local cache of model
About to load from disk!
About to Execute query!
INFO:llama_index.token_counter.token_counter:> [query] Total LLM token usage: 8165 tokens
INFO:llama_index.token_counter.token_counter:> [query] Total embedding token usage: 0 tokens
[2804.93106914 seconds]: f([]) ->
-
Source: Lloyd's List, 2023-03-31 02:51:48.205469
-
Source: Lloyd's List, 2023-03-31 02:51:48.205469
[NodeWithScore(node=Node(text='Summary for Australia\n\n- Source: The Interpreter, 2023-04-05 23:51:16.544469\n- Summary: Pressure is mounting on the Australian government to end public funding for international fossil fuel projects. Australia's major allies, including the US, the UK, Germany, France, and New Zealand, signed the Glasgow Statement last year, committing to ending public support for such projects. However, Australia has refused to commit to this so far. The country's financial support for fossil fuels abroad hasn't changed since May 2022, and the government continues to use taxpayer money to underwrite oil, gas, and coal projects. The Export Finance Australia (EFA), a public financial institution, provides low-interest loans and insurance coverage using government funds to Australian firms looking to offer goods and services overseas. \n- Keyword: Australia, fossil fuels, international funding, Glasgow Statement, taxpayer money, Export Finance Australia. \n- Read more: http://news.google.com/./articles/CBMiYWh0dHBzOi8vd3d3Lmxvd3lpbnN0aXR1dGUub3JnL3RoZS1pbnRlcnByZXRlci9hdXN0cmFsaWEtY2FuLW5vLWxvbmdlci1qdXN0aWZ5LWZvc3NpbC1mdWVsLWZ1bmRpbmfSAQA?hl=en-US&gl=US
System Specifications:
MacBook Pro - Apple M1 Pro - 16 GB - 240 GB disk storage free.
I also checked Activity monitor during the application execution- System 5%, User 20% idle - 75%
Thank you both for opening the issue. I’m getting help from some of my team members who have similar hardware specs to try and isolate the issue a bit further to see if I can reproduce.
That 2804 seconds is inexplicable even if you take away CUDA optimizations 😭
TypeError: CallbackManager.configure() got an unexpected keyword argument 'inheritable_tags'