xiph/vorbis

Two undefined behaviors by bitshifting of negative values

Opened this issue · 0 comments

Clang's sanitizers complain about bitshifting of negative values on 64-bit Ubuntu Linux:

1:

libvorbis/src/floor1.c:840:24: runtime error: left shift of negative value -2
#0 0x7f34f64c3eb1 in floor1_encode libvorbis/src/floor1.c
#1 0x7f34f650acd0 in mapping0_forward libvorbis/src/mapping0.c:625:20
#2 0x7f34f64b0035 in vorbis_analysis libvorbis/src/analysis.c:47:11

Fix:

libvorbis/src/floor1.c:

- val=-1-(val<<1);
+ val=-1-(val*2);

2:

libvorbis/src/psy.c:320:24: runtime error: left shift of negative value -11
#0 0x7fdfedcec8b5 in _vp_psy_init libvorbis/src/psy.c:320:24
#1 0x7fdfedcfc87b in _vds_shared_init libvorbis/src/block.c:225:7
#2 0x7fdfedcfbf6d in vorbis_analysis_init libvorbis/src/block.c:298:6

Fix:

libvorbis/src/psy.c:
- p->bark[i]=((lo-1)<<16)+(hi-1);
+ p->bark[i]=((lo-1)*65536)+(hi-1);