[Bug]: GPUExecutor throwing 'TypeError: 'type' object is not subscriptable' on 0.5.3
Closed this issue · 2 comments
xyzkpsf commented
Your current environment
Package Version
------------------------- --------------------
aiohttp 3.9.5
aiosignal 1.3.1
annotated-types 0.6.0
anyio 4.3.0
async-timeout 4.0.3
attrs 23.2.0
bitsandbytes 0.42.0
certifi 2019.11.28
chardet 3.0.4
charset-normalizer 3.3.2
click 8.1.7
cloudpickle 3.0.0
cmake 3.29.3
colorlog 6.8.2
dbus-python 1.2.16
diskcache 5.6.3
distro 1.9.0
dnspython 2.6.1
einops 0.8.0
email-validator 2.1.1
exceptiongroup 1.2.1
fastapi 0.111.0
fastapi-cli 0.0.3
filelock 3.14.0
frozenlist 1.4.1
fschat 0.2.36
fsspec 2024.3.1
h11 0.14.0
hf-transfer 0.1.6
httpcore 1.0.5
httpx 0.27.0
huggingface-hub 0.23.0
idna 2.8
importlib-metadata 7.1.0
importlib-resources 6.4.0
interegular 0.3.3
jinja2 3.1.4
joblib 1.4.2
jsonschema 4.22.0
jsonschema-specifications 2023.12.1
lark 1.1.8
llvmlite 0.41.1
lm-format-enforcer 0.10.1
loguru 0.7.2
markdown-it-py 3.0.0
markdown2 2.4.13
MarkupSafe 2.1.5
mdurl 0.1.2
mpmath 1.3.0
msgpack 1.0.8
multidict 6.0.5
nest-asyncio 1.6.0
networkx 3.1
nh3 0.2.17
ninja 1.11.1.1
numba 0.58.1
numpy 1.24.4
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu12 2.20.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu12 12.1.105
openai 1.29.0
orjson 3.10.3
outlines 0.0.41
packaging 24.0
pillow 10.3.0
pip 20.0.2
pkgutil-resolve-name 1.3.10
prometheus-client 0.20.0
prompt-toolkit 3.0.43
protobuf 5.26.1
psutil 5.9.8
pydantic 2.7.1
pydantic-core 2.18.2
pygments 2.18.0
PyGObject 3.36.0
pynvml 11.5.0
python-apt 2.0.1+ubuntu0.20.4.1
python-multipart 0.0.9
PyYAML 6.0.1
ray 2.10.0
referencing 0.35.1
regex 2024.5.10
requests 2.31.0
requests-unixsocket 0.2.0
rich 13.7.1
rpds-py 0.18.1
safetensors 0.4.3
scikit-learn 1.3.2
scipy 1.10.1
sentence-transformers 2.7.0
sentencepiece 0.2.0
setuptools 45.2.0
shellingham 1.5.4
shortuuid 1.0.13
six 1.14.0
sniffio 1.3.1
starlette 0.37.2
svgwrite 1.4.3
sympy 1.12
threadpoolctl 3.5.0
tiktoken 0.6.0
tokenizers 0.19.1
torch 2.3.0
tqdm 4.66.4
transformers 4.40.0
triton 2.3.0
typer 0.12.3
typing-extensions 4.11.0
ujson 5.10.0
urllib3 1.25.8
uvicorn 0.29.0
wavedrom 2.0.3.post3
wcwidth 0.2.13
wheel 0.34.2
xformers 0.0.26.post1
yarl 1.9.4
zipp 3.18.1
🐛 Describe the bug
i ran below code to create a simple api server:
python3 -m aphrodite.endpoints.openai.api_server \
--model microsoft/Phi-3-mini-4k-instruct \
--trust-remote-code
and got below error
/usr/local/lib/python3.8/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/root/aphrodite-engine/aphrodite/endpoints/openai/api_server.py", line 562, in <module>
run_server(args)
File "/root/aphrodite-engine/aphrodite/endpoints/openai/api_server.py", line 519, in run_server
engine = AsyncAphrodite.from_engine_args(engine_args)
File "/root/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 355, in from_engine_args
from aphrodite.executor.gpu_executor import GPUExecutorAsync
File "/root/aphrodite-engine/aphrodite/executor/gpu_executor.py", line 16, in <module>
class GPUExecutor(ExecutorBase):
File "/root/aphrodite-engine/aphrodite/executor/gpu_executor.py", line 92, in GPUExecutor
def determine_num_available_blocks(self) -> tuple[int, int]:
TypeError: 'type' object is not subscriptable
i tried install via both below but both got the error
$ pip install -U aphrodite-engine --extra-index-url https://downloads.pygmalion.chat/whl
or build from source
Not sure if that specific to my env?
sgsdxzy commented
The type annotation here is broken with Python 3.8. This is a bug, and before it is fixed can you upgrade your Python to 3.10 or 3.11?
xyzkpsf commented
yeah works after upgrading to python 3.11 thank you!