undefined symbol: _ZN2at4_ops5zeros4callEN3c108ArrayRefINS2_6SymIntEEENS2_8optionalINS2_10ScalarTypeEEENS6_INS2_6LayoutEEENS6_INS2_6DeviceEEENS6_IbEE

Oh yes i too have same issue

same here

I got the same issue. In my case I'm using Pytorch 2.2.2 (compiled with ABI) and CUDA 11.8. I tried flash attention V2 from the wheel and compiling from source. I got this error in both cases. @tridao would you know by any chance if latest version of flash attention support pytorch 2.2.2 (note that at the moment this is the latest release)

I haven't tried with pytorch 2.2.2 but I don't see why compiling from source wouldn't work. The wheel may or may not be compatible.

Hi @tridao thanks for your response.
I'm not too sure why it would fail. In my case I'm building flash-attention inside a docker container with the following commands:

ENV FLASH_ATTENTION_FORCE_BUILD=1
ENV FLASH_ATTENTION_FORCE_CXX11_ABI=1

RUN mkdir -p /build/flash_attention && cd /build/flash_attention \
  && git clone https://github.com/Dao-AILab/flash-attention -b 2.5.7 .

RUN cd /build/flash_attention \
  && python3 setup.py build bdist_wheel

The container where flash attention will be built has Pytorch 2.2.2 installed (With ABI).
Pytorch is correctly detected during the setup.py`:

dist.fetch_build_eggs(dist.setup_requires)

torch.__version__  = 2.2.2

The build will complete fine:

running bdist_wheel
Guessing wheel URL:  https://github.com/Dao-AILab/flash-attention/releases/download/v2.5.7/flash_attn-2.5.7+cu118torch2.2cxx11abiTRUE-cp310-cp310-linux_x86_64.whl
Raw wheel path dist/flash_attn-2.5.7-cp310-cp310-linux_x86_64.whl

Let me know if I can help with other testing. It looks like there are a few people with the same issue recently (see #723, #836 (comment), #854 and #821).

looks like it's still downloading the wheel? Can you try python3 setup.py install?

Thanks, I realised that ENV FLASH_ATTENTION_FORCE_BUILD=1 ENV FLASH_ATTENTION_FORCE_CXX11_ABI=1 were ignored because setup.py expect "TRUE" instead. This helped force rebuilding the wheel but the issue was something else.

In my case I had some python bindings that were using libtorch (the C++ pytorch library) but they were still using pointing to the older Pytorch version(2.1.2). Changing this to match the version used in Python did the trick.

Thanks for your help.

Might be unrelated but when using flash-attn 2.5.5 with torch 2.2.1 switching to py3.11 solved this issue for me.

We have new wheels (flash-attn 2.5.8) that should work with torch 2.2.2

It used to work until I try to install vllm. I met the same issue.

I just helped my colleague to fix this issue.

The reason is that flash attention 2.4.2 is using torch 2.2.0.dev20231106.

flash-attention/.github/workflows/publish.yml

Line 47 in 1a2c3e8

torch-version: ['1.12.1', '1.13.1', '2.0.1', '2.1.1', '2.2.0.dev20231106']

flash-attention/.github/workflows/publish.yml

Line 127 in 1a2c3e8

    
                       pip install --no-cache-dir --pre torch==${{ matrix.torch-version }} --index-url https://download.pytorch.org/whl/nightly/cu${TORCH_CUDA_VERSION}

i.e., a nightly built version of pytorch.

So if you also use a nightly built version, e.g.,
https://download.pytorch.org/whl/nightly/cu121/torch-2.2.0.dev20231010%2Bcu121-cp39-cp39-linux_x86_64.whl

then you should get rid of the symbol not found issue.

torch was using c10::optional in its nightly version but the released version swithes to std::optional; that is why so many of you are having this issue.

Thanks. Python 3.9.19，cuda12.2，torch2.3.0 flash_attn==2.5.8, it works.