Potential for integer overflow in `lovrModelAnimateVertices`
cjbarre opened this issue · 1 comments
cjbarre commented
Hi @bjornbytes,
Been corresponding on the lovr discord about the engine locking up when trying to load certain models on MacOS (M2).
I think I've isolated my current problem to this line.
On my system there's an integer overflow happening here which is causing the while loop that follows to be an infinite loop because the vertexCount
decrementer ends up being 0 due to the overflow.
When I set maxVerticesPerDispatch
to the max unint32 value as a test, then everything works as expected.
The numbers going into the calculation on my system end up being 1073741824 * 32 = 34359738368
.
I think the solution to this should probably include:
- Calculating that value differently and/or clamping to the max uint32 value.
- Adding a lovrAssert that either
maxVerticesPerDispatch
orvertexCount
are greater than zero