bjornbytes/lovr

Potential for integer overflow in `lovrModelAnimateVertices`

cjbarre opened this issue · 1 comments

Hi @bjornbytes,

Been corresponding on the lovr discord about the engine locking up when trying to load certain models on MacOS (M2).

I think I've isolated my current problem to this line.

On my system there's an integer overflow happening here which is causing the while loop that follows to be an infinite loop because the vertexCount decrementer ends up being 0 due to the overflow.

When I set maxVerticesPerDispatch to the max unint32 value as a test, then everything works as expected.

The numbers going into the calculation on my system end up being 1073741824 * 32 = 34359738368.

I think the solution to this should probably include:

  1. Calculating that value differently and/or clamping to the max uint32 value.
  2. Adding a lovrAssert that either maxVerticesPerDispatch or vertexCount are greater than zero

Here's a PR, feel free to use it, modify it, or toss it: #803