Since the b-frame patch landed dd604a0 the frame decode order is not the same as the frame display order. This needs to be fixed by adding code to decode the whole video (including caching all decctl metadata) and advancing it in the display order.
https://people.xiph.org/~unlord/nums.y4m