AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
vashiegaran opened this issue · 1 comments
I have tried to fix this problem by following all the solution in the Issues ex: downgrade CUDA
This is the library in the conda list , I need help to fix this
NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.2
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
absl-py 2.0.0 pypi_0 pypi
accelerate 0.25.0 pypi_0 pypi
aiofiles 23.2.1 pypi_0 pypi
aiohttp 3.9.1 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
altair 5.2.0 pypi_0 pypi
antlr4-python3-runtime 4.9.3 pypi_0 pypi
anyio 3.7.1 pypi_0 pypi
appdirs 1.4.4 pypi_0 pypi
async-timeout 4.0.3 pypi_0 pypi
attrs 23.1.0 pypi_0 pypi
bitsandbytes 0.38.1 pypi_0 pypi
blinker 1.7.0 pypi_0 pypi
ca-certificates 2023.12.12 h06a4308_0
certifi 2023.11.17 pypi_0 pypi
chardet 5.2.0 pypi_0 pypi
charset-normalizer 3.3.2 pypi_0 pypi
click 8.1.7 pypi_0 pypi
cmake 3.28.1 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
contourpy 1.2.0 pypi_0 pypi
cpm-kernels 1.0.11 pypi_0 pypi
cuda 11.7.0 0 nvidia/label/cuda-11.7.0
cuda-cccl 11.7.58 hc415cf5_0 nvidia/label/cuda-11.7.0
cuda-command-line-tools 11.7.0 0 nvidia/label/cuda-11.7.0
cuda-compiler 11.7.0 0 nvidia/label/cuda-11.7.0
cuda-cudart 11.7.60 h9538e0e_0 nvidia/label/cuda-11.7.0
cuda-cudart-dev 11.7.60 h6a7c232_0 nvidia/label/cuda-11.7.0
cuda-cuobjdump 11.7.50 h28cc80a_0 nvidia/label/cuda-11.7.0
cuda-cupti 11.7.50 hb6f9eaf_0 nvidia/label/cuda-11.7.0
cuda-cuxxfilt 11.7.50 hb365495_0 nvidia/label/cuda-11.7.0
cuda-demo-suite 11.7.50 0 nvidia/label/cuda-11.7.0
cuda-documentation 11.7.50 0 nvidia/label/cuda-11.7.0
cuda-driver-dev 11.7.60 0 nvidia/label/cuda-11.7.0
cuda-gdb 11.7.50 h4a0ac72_0 nvidia/label/cuda-11.7.0
cuda-libraries 11.7.0 0 nvidia/label/cuda-11.7.0
cuda-libraries-dev 11.7.0 0 nvidia/label/cuda-11.7.0
cuda-memcheck 11.7.50 hc446b2b_0 nvidia/label/cuda-11.7.0
cuda-nsight 11.7.50 0 nvidia/label/cuda-11.7.0
cuda-nsight-compute 11.7.0 0 nvidia/label/cuda-11.7.0
cuda-nvcc 11.7.64 0 nvidia/label/cuda-11.7.0
cuda-nvdisasm 11.7.50 h5bd0695_0 nvidia/label/cuda-11.7.0
cuda-nvml-dev 11.7.50 h3af1343_0 nvidia/label/cuda-11.7.0
cuda-nvprof 11.7.50 h7a2404d_0 nvidia/label/cuda-11.7.0
cuda-nvprune 11.7.50 h7add7b4_0 nvidia/label/cuda-11.7.0
cuda-nvrtc 11.7.50 hd0285e0_0 nvidia/label/cuda-11.7.0
cuda-nvrtc-dev 11.7.50 heada363_0 nvidia/label/cuda-11.7.0
cuda-nvtx 11.7.50 h05b0816_0 nvidia/label/cuda-11.7.0
cuda-nvvp 11.7.50 hd2289d5_0 nvidia/label/cuda-11.7.0
cuda-runtime 11.7.0 0 nvidia/label/cuda-11.7.0
cuda-sanitizer-api 11.7.50 hb424887_0 nvidia/label/cuda-11.7.0
cuda-toolkit 11.7.0 0 nvidia/label/cuda-11.7.0
cuda-tools 11.7.0 0 nvidia/label/cuda-11.7.0
cuda-visual-tools 11.7.0 0 nvidia/label/cuda-11.7.0
cycler 0.12.1 pypi_0 pypi
dataproperty 1.0.1 pypi_0 pypi
datasets 2.10.1 pypi_0 pypi
deepspeed 0.9.3 pypi_0 pypi
dill 0.3.4 pypi_0 pypi
distro 1.8.0 pypi_0 pypi
docker-pycreds 0.4.0 pypi_0 pypi
docstring-parser 0.15 pypi_0 pypi
evaluate 0.4.0 pypi_0 pypi
exceptiongroup 1.2.0 pypi_0 pypi
fastapi 0.105.0 pypi_0 pypi
ffmpy 0.3.1 pypi_0 pypi
filelock 3.13.1 pypi_0 pypi
flask 3.0.0 pypi_0 pypi
flask-cors 4.0.0 pypi_0 pypi
fonttools 4.46.0 pypi_0 pypi
frozenlist 1.4.0 pypi_0 pypi
fsspec 2023.9.2 pypi_0 pypi
gds-tools 1.3.0.44 0 nvidia/label/cuda-11.7.0
gitdb 4.0.11 pypi_0 pypi
gitpython 3.1.40 pypi_0 pypi
gradio 3.50.2 pypi_0 pypi
gradio-client 0.6.1 pypi_0 pypi
h11 0.14.0 pypi_0 pypi
hjson 3.1.0 pypi_0 pypi
httpcore 1.0.2 pypi_0 pypi
httpx 0.25.2 pypi_0 pypi
huggingface-hub 0.17.3 pypi_0 pypi
icetk 0.0.7 pypi_0 pypi
idna 3.6 pypi_0 pypi
importlib-metadata 7.0.0 pypi_0 pypi
importlib-resources 6.1.1 pypi_0 pypi
itsdangerous 2.1.2 pypi_0 pypi
jinja2 3.1.2 pypi_0 pypi
joblib 1.3.2 pypi_0 pypi
jsonlines 4.0.0 pypi_0 pypi
jsonschema 4.20.0 pypi_0 pypi
jsonschema-specifications 2023.11.2 pypi_0 pypi
kiwisolver 1.4.5 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1
libcublas 11.10.1.25 he442b6f_0 nvidia/label/cuda-11.7.0
libcublas-dev 11.10.1.25 h0c8ac2b_0 nvidia/label/cuda-11.7.0
libcufft 10.7.2.50 h80a1efe_0 nvidia/label/cuda-11.7.0
libcufft-dev 10.7.2.50 h59a5ac8_0 nvidia/label/cuda-11.7.0
libcufile 1.3.0.44 0 nvidia/label/cuda-11.7.0
libcufile-dev 1.3.0.44 0 nvidia/label/cuda-11.7.0
libcurand 10.2.10.50 heec50f7_0 nvidia/label/cuda-11.7.0
libcurand-dev 10.2.10.50 hd49a9cd_0 nvidia/label/cuda-11.7.0
libcusolver 11.3.5.50 hcab339c_0 nvidia/label/cuda-11.7.0
libcusolver-dev 11.3.5.50 hc6eba6f_0 nvidia/label/cuda-11.7.0
libcusparse 11.7.3.50 h6aaafad_0 nvidia/label/cuda-11.7.0
libcusparse-dev 11.7.3.50 hc644b96_0 nvidia/label/cuda-11.7.0
libffi 3.4.4 h6a678d5_0
libgcc-ng 11.2.0 h1234567_1
libgfortran-ng 7.5.0 ha8ba4b0_17
libgfortran4 7.5.0 ha8ba4b0_17
libgomp 11.2.0 h1234567_1
libnpp 11.7.3.21 h3effbd9_0 nvidia/label/cuda-11.7.0
libnpp-dev 11.7.3.21 hb6476a9_0 nvidia/label/cuda-11.7.0
libnvjpeg 11.7.2.34 hfe236c7_0 nvidia/label/cuda-11.7.0
libnvjpeg-dev 11.7.2.34 h2e48410_0 nvidia/label/cuda-11.7.0
libstdcxx-ng 11.2.0 h1234567_1
lit 17.0.6 pypi_0 pypi
lm-eval 0.3.0 pypi_0 pypi
lmflow 0.0.1 dev_0
markdown-it-py 3.0.0 pypi_0 pypi
markupsafe 2.1.3 pypi_0 pypi
matplotlib 3.8.2 pypi_0 pypi
mbstrdecoder 1.1.3 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
mpi 1.0.0 pypi_0 pypi
mpi4py 3.1.4 py39hfc96bbd_0
mpich 3.3.2 hc856adb_0
mpmath 1.3.0 pypi_0 pypi
multidict 6.0.4 pypi_0 pypi
multiprocess 0.70.12.2 pypi_0 pypi
ncurses 6.4 h6a678d5_0
networkx 3.2.1 pypi_0 pypi
ninja 1.11.1.1 pypi_0 pypi
nltk 3.8.1 pypi_0 pypi
nsight-compute 2022.2.0.13 0 nvidia/label/cuda-11.7.0
numexpr 2.8.8 pypi_0 pypi
numpy 1.24.2 pypi_0 pypi
nvidia-cublas-cu11 11.10.3.66 pypi_0 pypi
nvidia-cuda-cupti-cu11 11.7.101 pypi_0 pypi
nvidia-cuda-nvrtc-cu11 11.7.99 pypi_0 pypi
nvidia-cuda-runtime-cu11 11.7.99 pypi_0 pypi
nvidia-cudnn-cu11 8.5.0.96 pypi_0 pypi
nvidia-cufft-cu11 10.9.0.58 pypi_0 pypi
nvidia-curand-cu11 10.2.10.91 pypi_0 pypi
nvidia-cusolver-cu11 11.4.0.1 pypi_0 pypi
nvidia-cusparse-cu11 11.7.4.91 pypi_0 pypi
nvidia-nccl-cu11 2.14.3 pypi_0 pypi
nvidia-nvtx-cu11 11.7.91 pypi_0 pypi
omegaconf 2.3.0 pypi_0 pypi
openai 1.3.8 pypi_0 pypi
openssl 3.0.12 h7f8727e_0
orjson 3.9.10 pypi_0 pypi
packaging 23.2 pypi_0 pypi
pandas 2.1.4 pypi_0 pypi
pathtools 0.1.2 pypi_0 pypi
pathvalidate 3.2.0 pypi_0 pypi
peft 0.3.0.dev0 pypi_0 pypi
pillow 10.1.0 pypi_0 pypi
pip 23.3.1 py39h06a4308_0
portalocker 2.8.2 pypi_0 pypi
protobuf 3.18.3 pypi_0 pypi
psutil 5.9.6 pypi_0 pypi
py-cpuinfo 9.0.0 pypi_0 pypi
pyarrow 14.0.1 pypi_0 pypi
pyarrow-hotfix 0.6 pypi_0 pypi
pybind11 2.11.1 pypi_0 pypi
pycountry 23.12.11 pypi_0 pypi
pydantic 1.10.9 pypi_0 pypi
pydub 0.25.1 pypi_0 pypi
pygments 2.17.2 pypi_0 pypi
pyparsing 3.1.1 pypi_0 pypi
pytablewriter 1.2.0 pypi_0 pypi
python 3.9.18 h955ad1f_0
python-dateutil 2.8.2 pypi_0 pypi
python-multipart 0.0.6 pypi_0 pypi
pytz 2023.3.post1 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
readline 8.2 h5eee18b_0
referencing 0.32.0 pypi_0 pypi
regex 2023.10.3 pypi_0 pypi
requests 2.31.0 pypi_0 pypi
responses 0.18.0 pypi_0 pypi
rich 13.7.0 pypi_0 pypi
rouge-score 0.1.2 pypi_0 pypi
rpds-py 0.15.2 pypi_0 pypi
sacrebleu 1.5.0 pypi_0 pypi
safetensors 0.4.1 pypi_0 pypi
scikit-learn 1.2.2 pypi_0 pypi
scipy 1.11.4 pypi_0 pypi
semantic-version 2.10.0 pypi_0 pypi
sentencepiece 0.1.99 pypi_0 pypi
sentry-sdk 1.38.0 pypi_0 pypi
setproctitle 1.3.3 pypi_0 pypi
setuptools 68.0.0 pypi_0 pypi
shtab 1.6.5 pypi_0 pypi
six 1.16.0 pypi_0 pypi
smmap 5.0.1 pypi_0 pypi
sniffio 1.3.0 pypi_0 pypi
sqlite 3.41.2 h5eee18b_0
sqlitedict 2.1.0 pypi_0 pypi
starlette 0.27.0 pypi_0 pypi
sympy 1.12 pypi_0 pypi
tabledata 1.3.3 pypi_0 pypi
tcolorpy 0.1.4 pypi_0 pypi
threadpoolctl 3.2.0 pypi_0 pypi
tk 8.6.12 h1ccaba5_0
tokenizers 0.14.1 pypi_0 pypi
toolz 0.12.0 pypi_0 pypi
torch 2.0.0 pypi_0 pypi
torchvision 0.15.1 pypi_0 pypi
tqdm 4.66.1 pypi_0 pypi
tqdm-multiprocess 0.0.11 pypi_0 pypi
transformers 4.34.0 pypi_0 pypi
triton 2.0.0 pypi_0 pypi
trl 0.7.5.dev0 pypi_0 pypi
typepy 1.3.2 pypi_0 pypi
typing-extensions 4.9.0 pypi_0 pypi
tyro 0.6.0 pypi_0 pypi
tzdata 2023.3 pypi_0 pypi
urllib3 2.1.0 pypi_0 pypi
uvicorn 0.24.0.post1 pypi_0 pypi
wandb 0.14.0 pypi_0 pypi
websockets 11.0.3 pypi_0 pypi
werkzeug 3.0.1 pypi_0 pypi
wheel 0.41.2 py39h06a4308_0
xxhash 3.4.1 pypi_0 pypi
xz 5.4.5 h5eee18b_0
yarl 1.9.4 pypi_0 pypi
zipp 3.17.0 pypi_0 pypi
zlib 1.2.13 h5eee18b_0
zstandard 0.22.0 pypi_0 pypi
Thanks for your interest in LMFlow! One source of this problem can be the compilation failures of CPU adam operator, where deepspeed requires a nvcc/c++ compilation process to support running Adam in CPUs.
We are wondering if you could provide the error logs so we could help you check the specific reason? Thanks very much 😄