Speed changes with scaletempo2 can cause audio/video desynchronization

[ferreum]

Important Information

• mpv version: 0.36.0, also observed in 0.35.0 and current git at bc96b23
• Arch Linux, up-to-date
• installed from arch package repositories / AUR

Reproduction steps

1. Find a video where audio/video sync can be observed well (e.g. https://www.youtube.com/watch?v=jpKrrchVRGE)
2. Open the video with mpv --no-config --speed=1.01 <url>
3. Quickly change the playback speed back and forth between 1.01x and 2.02x using then } and { bindings. I had success by switching back and forth around twice a second or more, that is, at least 4 key presses a second.
4. Do this for a while as the video plays. Make sure speed does not go to 1x, as that resets the scaletempo2 filter. Do not seek the video, as that resets playback too.
5. After doing this for about 30-50 seconds of the video, let it play at 1.01x and check the a/v sync.

Expected behavior

Audio/Video stays synchronized. In the example video above, the numbers are spoken exactly when they first appear.
This does work when specifying --af=scaletempo or --af=rubberband.

Actual behavior

Audio is played back ahead of the video. In the example video above, the numbers are spoken before they appear, easily by 0.5s, but 1s or more is possible.

Note: when reverting to 1x speed, the scaletempo2 filter is removed, which fixes the desync. There are different ways this plays out:

• sometimes, the video is forwarded to the correct time, catching up to the audio position
• other times, the audio cuts out or repeats a short moment while the video plays normally

Log file

mpv-log.txt

Sample files

So far I could reproduce this in any video I want. See the countdown video linked above for easy tests.

Additional info

The scaletempo2 code went over my head, but I tried changing random things. I was lucky to find that removing this piece of code fixes the desynchronization:

diff --git audio/filter/af_scaletempo2.c audio/filter/af_scaletempo2.c
index 1a822ecd50..bbed7d3e9d 100644
--- audio/filter/af_scaletempo2.c
+++ audio/filter/af_scaletempo2.c
@@ -111,11 +111,6 @@ static void process(struct mp_filter *f)
         double pts = mp_aframe_get_pts(p->pending);
         p->frame_delay -= out_samples * p->speed;

-        if (pts != MP_NOPTS_VALUE) {
-            double delay = p->frame_delay / mp_aframe_get_effective_rate(out);
-            mp_aframe_set_pts(out, pts - delay);
-        }
-
         mp_aframe_set_size(out, out_samples);
         mp_aframe_mul_speed(out, p->speed);
         mp_pin_in_write(f->ppins[1], MAKE_FRAME(MP_FRAME_AUDIO, out));

I assume removing this code would break other situations, but I hope this helps someone find the problem.

Relates to #6797 and my script https://github.com/ferreum/mpv-skipsilence, where the frequent speed changes cause this problem too.

[christoph-heinrich]

Can reproduce when using the linked script with linked video with pretty aggressive settings (ramp_constant = 4 to immediately hit max speed). It slowly de-syncs over time, I have to watch 10s+ for it to become very noticeable.

How quickly it gets out of sync depends on video-sync, but it ends up de-syncing with all of them.

P.S. Please add that script to the wiki. Hopefully it will eventually become as good as the feature in NewPipe.