Stack overflow in AVX2 code

[thomwiggers]

Compiling test_kyber512 with -fsanitize=address,undefined on gcc 10.1.0 leads to the following:

=================================================================
==288165==ERROR: AddressSanitizer: global-buffer-overflow on address 0x55b880d170b8 at pc 0x55b880c9b0fc bp 0x7fff1a3c8950 sp 0x7fff1a3c8940
READ of size 16 at 0x55b880d170b8 thread T0
    #0 0x55b880c9b0fb in pqcrystals_kyber512_avx2_rej_uniform_avx (/home/thom/git/phd/kyber/avx2/test_kyber512+0x710fb)
    #1 0x55b880c897c6 in pqcrystals_kyber512_avx2_gen_matrix (/home/thom/git/phd/kyber/avx2/test_kyber512+0x5f7c6)
    #2 0x55b880c8b5e0 in pqcrystals_kyber512_avx2_indcpa_keypair (/home/thom/git/phd/kyber/avx2/test_kyber512+0x615e0)
    #3 0x55b880c88a61 in pqcrystals_kyber512_avx2_keypair (/home/thom/git/phd/kyber/avx2/test_kyber512+0x5ea61)
    #4 0x55b880d13b9a in test_keys.isra.0 (/home/thom/git/phd/kyber/avx2/test_kyber512+0xe9b9a)
    #5 0x55b880c888ba in main (/home/thom/git/phd/kyber/avx2/test_kyber512+0x5e8ba)
    #6 0x7f366960a001 in __libc_start_main (/usr/lib/libc.so.6+0x27001)
    #7 0x55b880c8896d in _start (/home/thom/git/phd/kyber/avx2/test_kyber512+0x5e96d)

0x55b880d170c0 is located 0 bytes to the right of global variable 'idx' defined in 'rejsample.c:8:22' (0x55b880d168c0) of size 2048
SUMMARY: AddressSanitizer: global-buffer-overflow (/home/thom/git/phd/kyber/avx2/test_kyber512+0x710fb) in pqcrystals_kyber512_avx2_rej_uniform_avx
Shadow bytes around the buggy address:
  0x0ab79019adc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ab79019add0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ab79019ade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ab79019adf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ab79019ae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0ab79019ae10: 00 00 00 00 00 00 00[00]f9 f9 f9 f9 00 00 00 00
  0x0ab79019ae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ab79019ae30: 00 00 00 04 f9 f9 f9 f9 06 f9 f9 f9 f9 f9 f9 f9
  0x0ab79019ae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 01 f9 f9
  0x0ab79019ae50: f9 f9 f9 f9 00 00 00 00 00 00 00 04 f9 f9 f9 f9
  0x0ab79019ae60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==288165==ABORTING

[gregorseiler]

Thanks! The problem was using _mm_loadu_si128 instead of _mm_loadl_epi64 to read 8 bytes from the idx table. It is fixed now.