4 Failing `test_flash_attn_output_fp8` tests on H100
BioGeek opened this issue · 3 comments
BioGeek commented
I installed Flash Attention 3 on a system with H100 GPUs and ran the tests. 12436 passed, but 4 failed:
FAILED hopper/test_flash_attn.py::test_flash_attn_output_fp8[1024-1023-1.0-128-False-True-False-True-gqa-dtype0]
FAILED hopper/test_flash_attn.py::test_flash_attn_output_fp8[1024-1023-1.0-128-False-True-True-True-gqa-dtype0]
FAILED hopper/test_flash_attn.py::test_flash_attn_output_fp8[1024-1023-1.0-128-True-True-False-True-gqa-dtype0]
FAILED hopper/test_flash_attn.py::test_flash_attn_output_fp8[1024-1023-1.0-128-True-True-True-True-gqa-dtype0]
The differences is small, so could be a random fluke?
E Mismatched elements: 1 / 3145728 (0.0%)
E Greatest absolute difference: 0.08984375 at index (3, 3, 0, 80) (up to 0.0725 allowed)
E Greatest relative difference: 0.1435546875 at index (3, 3, 0, 80) (up to 0.01 allowed)
Full log is attached: experiment.log
robogast commented
Same here:
================================================================================== short test summary info ===================================================================================
FAILED test_flash_attn.py::test_flash_attn_output_fp8[1024-1023-1.0-128-False-True-False-True-gqa-dtype0] - AssertionError: Tensor-likes are not close!
FAILED test_flash_attn.py::test_flash_attn_output_fp8[1024-1023-1.0-128-False-True-True-True-gqa-dtype0] - AssertionError: Tensor-likes are not close!
FAILED test_flash_attn.py::test_flash_attn_output_fp8[1024-1023-1.0-128-True-True-False-True-gqa-dtype0] - AssertionError: Tensor-likes are not close!
FAILED test_flash_attn.py::test_flash_attn_output_fp8[1024-1023-1.0-128-True-True-True-True-gqa-dtype0] - AssertionError: Tensor-likes are not close!
4 failed, 12436 passed in 125.88s (0:02:05)
jyjfjyjf commented