tag-mismatch (HWAsan) triggered by cffi

[n-bes]

Hi, i found few issues in projects which uses cffi:

• sde1000/python-xkbcommon#24
• pyca/cryptography#11217

Could you comment it?

[n-bes]

Catch without dependencies

Dockerfile:

FROM ubuntu:24.04 as ubuntu_env
RUN apt-get update -y && \
    apt-get install -y \
        autoconf \
        clang \
        clang-tools \
        curl \
        gcc \
        lld \
        pkg-config \
        python3-dbg \
        python3-dev \
        python3-pip \
        python3-venv
ENV PATH=/root/.cargo/bin:$PATH \
    CC=clang \
    CXX=clang++ \
    CFLAGS="-g -O0 -fsanitize=hwaddress -fuse-ld=lld -g" \
    CCFLAGS="-g -O0 -fsanitize=hwaddress -fuse-ld=lld -g" \
    CXXFLAGS="-g -O0 -fsanitize=hwaddress -fuse-ld=lld -g" \
    CPPFLAGS="-g -O0 -fsanitize=hwaddress -fuse-ld=lld -g" \
    LDFLAGS="-fsanitize=hwaddress -fuse-ld=lld" \
    LD_PRELOAD="/usr/lib/llvm-18/lib/clang/18/lib/linux/libclang_rt.hwasan-aarch64.so" \
    ASAN_OPTIONS="detect_leaks=0" \
    HWASAN_OPTIONS="detect_leaks=0"

$ docker run --rm -it <image>
$ python3 -m venv venv
$ source venv/bin/activate
$ pip3 install cffi --no-binary ":all:"
Collecting cffi
  Downloading cffi-1.16.0.tar.gz (512 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 512.9/512.9 kB 4.9 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pycparser (from cffi)
  Downloading pycparser-2.22.tar.gz (172 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 172.7/172.7 kB 8.6 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: cffi, pycparser
  Building wheel for cffi (pyproject.toml) ... done
  Created wheel for cffi: filename=cffi-1.16.0-cp312-cp312-linux_aarch64.whl size=558608 sha256=415117390fdbb7d1635510da09f6bedea8230e6da04bdfc448ea5b5aaf9bdf38
  Stored in directory: /root/.cache/pip/wheels/9f/20/b5/43b003d703a3519a9adbe40423a752dd0c7618afb763739963
  Building wheel for pycparser (pyproject.toml) ... done
  Created wheel for pycparser: filename=pycparser-2.22-py3-none-any.whl size=117551 sha256=b84c737cf24cd6d305dd0b287ed4bdb65f329061324e027867220149457e1e6b
  Stored in directory: /root/.cache/pip/wheels/36/53/17/c0ae2e096d359a9a8faf47fd7ded8f4c878af41a3c66cb5199
Successfully built cffi pycparser
Installing collected packages: pycparser, cffi
Successfully installed cffi-1.16.0 pycparser-2.22

$ python3
Python 3.12.3 (main, Apr 10 2024, 05:33:47) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import _cffi_backend

==83==ERROR: HWAddressSanitizer: tag-mismatch on address 0xffff9aa38560 at pc 0xffff9a91640c
READ of size 8 at 0xffff9aa38560 tags: b0/00 (ptr/mem) in thread T0
    #0 0xffff9a91640c in PyInit__cffi_backend /tmp/pip-install-yq1ib0wm/cffi_5a5f2e7dfea04f2eaa284d202c93422c/src/c/_cffi_backend.c:8038:9
    #1 0x66def0 in _PyImport_LoadDynamicModuleWithSpec /usr/src/python3.12-3.12.3-1/build-static/../Python/importdl.c:169:9
    #2 0x66d338 in _imp_create_dynamic_impl /usr/src/python3.12-3.12.3-1/build-static/../Python/import.c:3775:11
    #3 0x66d338 in _imp_create_dynamic /usr/src/python3.12-3.12.3-1/build-static/../Python/clinic/import.c.h:506:20
    #4 0x502d4c in cfunction_vectorcall_FASTCALL /usr/src/python3.12-3.12.3-1/build-static/../Objects/methodobject.c:422:24
    #5 0x5652ac in _PyEval_EvalFrameDefault /usr/src/python3.12-3.12.3-1/build-static/Python/bytecodes.c:3254:26
    #6 0x4c2f00 in _PyObject_VectorcallTstate /usr/src/python3.12-3.12.3-1/build-static/../Include/internal/pycore_call.h:92:11
    #7 0x4c2f00 in object_vacall /usr/src/python3.12-3.12.3-1/build-static/../Objects/call.c:850:14
    #8 0x4c4ae4 in PyObject_CallMethodObjArgs /usr/src/python3.12-3.12.3-1/build-static/../Objects/call.c:911:24
    #9 0x58bee8 in import_find_and_load /usr/src/python3.12-3.12.3-1/build-static/../Python/import.c:2779:11
    #10 0x58bee8 in PyImport_ImportModuleLevelObject /usr/src/python3.12-3.12.3-1/build-static/../Python/import.c:2862:15
    #11 0x565f6c in import_name /usr/src/python3.12-3.12.3-1/build-static/../Python/ceval.c:2482:15
    #12 0x565f6c in _PyEval_EvalFrameDefault /usr/src/python3.12-3.12.3-1/build-static/Python/bytecodes.c:2135:19
    #13 0x560070 in _PyEval_EvalFrame /usr/src/python3.12-3.12.3-1/build-static/../Include/internal/pycore_ceval.h:89:16
    #14 0x560070 in _PyEval_Vector /usr/src/python3.12-3.12.3-1/build-static/../Python/ceval.c:1683:12
    #15 0x560070 in PyEval_EvalCode /usr/src/python3.12-3.12.3-1/build-static/../Python/ceval.c:578:21
    #16 0x598f70 in run_eval_code_obj /usr/src/python3.12-3.12.3-1/build-static/../Python/pythonrun.c:1722:9
    #17 0x598f70 in run_mod /usr/src/python3.12-3.12.3-1/build-static/../Python/pythonrun.c:1743:19
    #18 0x451460 in PyRun_InteractiveOneObjectEx /usr/src/python3.12-3.12.3-1/build-static/../Python/pythonrun.c:260:9
    #19 0x4511a8 in _PyRun_InteractiveLoopObject /usr/src/python3.12-3.12.3-1/build-static/../Python/pythonrun.c:137:15
    #20 0x67e0e4 in _PyRun_AnyFileObject /usr/src/python3.12-3.12.3-1/build-static/../Python/pythonrun.c:72:15
    #21 0x451078 in PyRun_AnyFileExFlags /usr/src/python3.12-3.12.3-1/build-static/../Python/pythonrun.c:104:15
    #22 0x689500 in pymain_run_stdin /usr/src/python3.12-3.12.3-1/build-static/../Modules/main.c:520:15
    #23 0x689500 in pymain_run_python /usr/src/python3.12-3.12.3-1/build-static/../Modules/main.c:632:21
    #24 0x689500 in Py_RunMain /usr/src/python3.12-3.12.3-1/build-static/../Modules/main.c:709:5
    #25 0x688ca4 in Py_BytesMain /usr/src/python3.12-3.12.3-1/build-static/../Modules/main.c:763:12
    #26 0xffff9c9684c0 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #27 0xffff9c968594 in __libc_start_main csu/../csu/libc-start.c:360:3
    #28 0x5f24ec in _start (/usr/bin/python3.12+0x5f24ec) (BuildId: 18160fe6beb052a7e6830ecc99e313a3498c377d)


Thread: T0 0xeffe00002000 stack: [0xffffe28ab000,0xffffe30ab000) sz: 8388608 tls: [0xffff9d720460,0xffff9d721320)

Memory tags around the buggy address (one tag corresponds to 16 bytes):
  0xffff9aa37d00: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff9aa37e00: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff9aa37f00: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff9aa38000: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff9aa38100: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff9aa38200: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff9aa38300: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff9aa38400: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
=>0xffff9aa38500: 00  00  00  00  00  00 [00] 00  00  00  00  00  00  00  00  00
  0xffff9aa38600: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff9aa38700: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff9aa38800: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff9aa38900: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff9aa38a00: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff9aa38b00: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff9aa38c00: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff9aa38d00: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
Tags for short granules around the buggy address (one tag corresponds to 16 bytes):
  0xffff9aa38400: ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..
=>0xffff9aa38500: ..  ..  ..  ..  ..  .. [..] ..  ..  ..  ..  ..  ..  ..  ..  ..
  0xffff9aa38600: ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..
See https://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html#short-granules for a description of short granule tags

Registers where the failure occurred (pc 0xffff9a91640c):
    x0  b000ffff9aa38560  x1  0000000000000000  x2  0000000000000000  x3  0000000000000001
    x4  0000000000000002  x5  8400ed5e0000bb00  x6  0000000000000006  x7  00000000000000a0
    x8  00000000000000b0  x9  0000000000000000  x10 00000ffff9aa3856  x11 0000000000000000
    x12 0000000000000000  x13 0000000000000084  x14 0000000000000000  x15 0000ecebfffe0024
    x16 0000ffff9cc6b7f0  x17 0000000000000007  x18 0000000000000004  x19 0000ffff9ac074f0
    x20 0200efff00000000  x21 0000ffff9aa72f30  x22 0000ffff9aab3240  x23 0000000000000000
    x24 0000ffff9aab32f0  x25 000000000070f5b8  x26 0000ffff9a915fb4  x27 0000ffff9aab3260
    x28 0000000000b8a278  x29 0000ffffe30a9760  x30 0000ffff9a916410   sp 0000ffffe30a95e0
SUMMARY: HWAddressSanitizer: tag-mismatch /tmp/pip-install-yq1ib0wm/cffi_5a5f2e7dfea04f2eaa284d202c93422c/src/c/_cffi_backend.c:8038:9 in PyInit__cffi_backend

[arigo]

I have no clue about HWAsan. I guess it's an address-sanitizer and it crashes on the line mentioned in the other issue (https://github.com/python-cffi/cffi/blob/v1.16.0/src/c/_cffi_backend.c#L8038). My only clue would be that it complains that NULL might in theory be different from all-zeroes, which is the initial value of the pointer. While that's correct in theory, and could be fixed here, I also think it's kinda pointless. On all architectures, NULL is the all-zero value, since decades ago.

[n-bes]

I think that unique_cache is simply not initialized:

cffi/src/c/_cffi_backend.c

Line 388 in ba44abd

static PyObject *unique_cache;

I guess it's an address-sanitizer

Yes, but hardware implementation (arm-mte).

[alex]

Static globals are always (by specification) 0-initialized, I believe?

…

On Tue, Jul 9, 2024, 11:38 AM n-bes ***@***.***> wrote: I think that unique_cache just is not uninitialized: https://github.com/python-cffi/cffi/blob/ba44abd69cf6f0f1cc90db34cd067275dc10fc71/src/c/_cffi_backend.c#L388 I guess it's an address-sanitizer Yes, but hardware implementation (arm-mte). — Reply to this email directly, view it on GitHub <#93 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBBSTZJXLIIA32RU26TZLP7YJAVCNFSM6AAAAABKSRQBQSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJYGA2DEMJZGY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[arigo]

It is initialized. The C standard says it is initialized to all-zeroes. The part outside the C standard is that all-zeroes happens to be the NULL value on all platforms since forever.

[arigo]

Sorry if that wasn't clear: try to initialize the value to NULL explicitly:

static PyObject *unique_cache = NULL; 

It shouldn't change anything at all, except make the address sanitizer happy?

[n-bes]

try to initialize the value to NULL explicitly:

Nothing changed at all:

387	static PyObject *FFIError = NULL;
388	static PyObject *unique_cache = NULL;

>>> import _cffi_backend
==75==ERROR: HWAddressSanitizer: tag-mismatch on address 0xffff7d9c8560 at pc 0xffff7d8a640c
READ of size 8 at 0xffff7d9c8560 tags: b0/00 (ptr/mem) in thread T0
    #0 0xffff7d8a640c in PyInit__cffi_backend /src/src/c/_cffi_backend.c:8038:9
    #1 0x66def0 in _PyImport_LoadDynamicModuleWithSpec /usr/src/python3.12-3.12.3-1/build-static/../Python/importdl.c:169:9
    #2 0x66d338 in _imp_create_dynamic_impl /usr/src/python3.12-3.12.3-1/build-static/../Python/import.c:3775:11
    #3 0x66d338 in _imp_create_dynamic /usr/src/python3.12-3.12.3-1/build-static/../Python/clinic/import.c.h:506:20
    #4 0x502d4c in cfunction_vectorcall_FASTCALL /usr/src/python3.12-3.12.3-1/build-static/../Objects/methodobject.c:422:24
    #5 0x5652ac in _PyEval_EvalFrameDefault /usr/src/python3.12-3.12.3-1/build-static/Python/bytecodes.c:3254:26
    #6 0x4c2f00 in _PyObject_VectorcallTstate /usr/src/python3.12-3.12.3-1/build-static/../Include/internal/pycore_call.h:92:11
    #7 0x4c2f00 in object_vacall /usr/src/python3.12-3.12.3-1/build-static/../Objects/call.c:850:14
    #8 0x4c4ae4 in PyObject_CallMethodObjArgs /usr/src/python3.12-3.12.3-1/build-static/../Objects/call.c:911:24
    #9 0x58bee8 in import_find_and_load /usr/src/python3.12-3.12.3-1/build-static/../Python/import.c:2779:11
    #10 0x58bee8 in PyImport_ImportModuleLevelObject /usr/src/python3.12-3.12.3-1/build-static/../Python/import.c:2862:15
    #11 0x565f6c in import_name /usr/src/python3.12-3.12.3-1/build-static/../Python/ceval.c:2482:15
    #12 0x565f6c in _PyEval_EvalFrameDefault /usr/src/python3.12-3.12.3-1/build-static/Python/bytecodes.c:2135:19
    #13 0x560070 in _PyEval_EvalFrame /usr/src/python3.12-3.12.3-1/build-static/../Include/internal/pycore_ceval.h:89:16
    #14 0x560070 in _PyEval_Vector /usr/src/python3.12-3.12.3-1/build-static/../Python/ceval.c:1683:12
    #15 0x560070 in PyEval_EvalCode /usr/src/python3.12-3.12.3-1/build-static/../Python/ceval.c:578:21
    #16 0x598f70 in run_eval_code_obj /usr/src/python3.12-3.12.3-1/build-static/../Python/pythonrun.c:1722:9
    #17 0x598f70 in run_mod /usr/src/python3.12-3.12.3-1/build-static/../Python/pythonrun.c:1743:19
    #18 0x451460 in PyRun_InteractiveOneObjectEx /usr/src/python3.12-3.12.3-1/build-static/../Python/pythonrun.c:260:9
    #19 0x4511a8 in _PyRun_InteractiveLoopObject /usr/src/python3.12-3.12.3-1/build-static/../Python/pythonrun.c:137:15
    #20 0x67e0e4 in _PyRun_AnyFileObject /usr/src/python3.12-3.12.3-1/build-static/../Python/pythonrun.c:72:15
    #21 0x451078 in PyRun_AnyFileExFlags /usr/src/python3.12-3.12.3-1/build-static/../Python/pythonrun.c:104:15
    #22 0x689500 in pymain_run_stdin /usr/src/python3.12-3.12.3-1/build-static/../Modules/main.c:520:15
    #23 0x689500 in pymain_run_python /usr/src/python3.12-3.12.3-1/build-static/../Modules/main.c:632:21
    #24 0x689500 in Py_RunMain /usr/src/python3.12-3.12.3-1/build-static/../Modules/main.c:709:5
    #25 0x688ca4 in Py_BytesMain /usr/src/python3.12-3.12.3-1/build-static/../Modules/main.c:763:12
    #26 0xffff7f9184c0 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #27 0xffff7f918594 in __libc_start_main csu/../csu/libc-start.c:360:3
    #28 0x5f24ec in _start (/usr/bin/python3.12+0x5f24ec) (BuildId: 18160fe6beb052a7e6830ecc99e313a3498c377d)


Thread: T0 0xeffe00002000 stack: [0xffffc4383000,0xffffc4b83000) sz: 8388608 tls: [0xffff806d0460,0xffff806d1320)

Memory tags around the buggy address (one tag corresponds to 16 bytes):
  0xffff7d9c7d00: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff7d9c7e00: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff7d9c7f00: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff7d9c8000: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff7d9c8100: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff7d9c8200: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff7d9c8300: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff7d9c8400: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
=>0xffff7d9c8500: 00  00  00  00  00  00 [00] 00  00  00  00  00  00  00  00  00
  0xffff7d9c8600: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff7d9c8700: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff7d9c8800: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff7d9c8900: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff7d9c8a00: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff7d9c8b00: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff7d9c8c00: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
  0xffff7d9c8d00: 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
Tags for short granules around the buggy address (one tag corresponds to 16 bytes):
  0xffff7d9c8400: ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..
=>0xffff7d9c8500: ..  ..  ..  ..  ..  .. [..] ..  ..  ..  ..  ..  ..  ..  ..  ..
  0xffff7d9c8600: ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..
See https://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html#short-granules for a description of short granule tags

Registers where the failure occurred (pc 0xffff7d8a640c):
    x0  b000ffff7d9c8560  x1  0000000000000000  x2  0000000000000000  x3  0000000000000001
    x4  0000000000000002  x5  9600ed5e0000bb00  x6  0000000000000006  x7  0000000000000050
    x8  00000000000000b0  x9  0000000000000000  x10 00000ffff7d9c856  x11 0000000000000000
    x12 0000000000000000  x13 0000000000000096  x14 0000000000000000  x15 0000ecebfffe0024
    x16 0000ffff7fc1b7f0  x17 0000000000000007  x18 0000000000000004  x19 0000ffff7dbc7830
    x20 0200efff00000000  x21 0000ffff7d9efe10  x22 0000ffff7eeb3b40  x23 0000000000000000
    x24 0000ffff7da474d0  x25 000000000070f5b8  x26 0000ffff7d8a5fb4  x27 0000ffff7eeb3b60
    x28 0000000000b8a278  x29 0000ffffc4b80d70  x30 0000ffff7d8a6410   sp 0000ffffc4b80bf0
SUMMARY: HWAddressSanitizer: tag-mismatch /src/src/c/_cffi_backend.c:8038:9 in PyInit__cffi_backend

[arigo]

Sorry, no idea. Try to look for help in places related to HWAsan.

[alex]

This looks an awful lot like google/sanitizers#1245, which seems to suggest this may be an hwasan bug.

[n-bes]

@alex, good link. Stack-trace looks like a false-positive, I'll rebuild with additional flags.