Vp8 Adaptive Projection and gstreamer - motion artifacts

[jpietek]

I'm trying to use jvb simulcast - adaptive vp8 projection with gstreamer, thanks to a wonderful gst-meet tool, the original issue described here: avstack/gst-meet#22

Current behavior

After turning simulcast on at JVB, the gstreamer rtpjitterbuffer/vp8depayloader reports discontinuities. Motion artifacts are clearly visible after decoding. This happens when jitter and packet loss occurs between sender and jvb. The same vp8 projection output is fine inside the browser, it drops frames, but no artifacts. The jicofo is set to stripSimulcast = true and jvb rewrites all the packets into single low layer ssrc, so I believe full simulcast understanding is not required for gstreamer.

Expected Behavior

The jvb output is continuous after NACKs, thus the gstreamer's vp8depayloader feeds the decoder with decode-able vp8 frames.

Possible Solution

I believe the problem is with retransmissions signalled by the gstreamer jitter buffer. NACKs between sender and jvb are not enough. When I turn the "do-retransmissions" property to false, gstreamer is having artifacts without simulcast too. Turning RTX on - fixes the video for GenericVp8Projection.

Maybe there is a way to rewriteRtcp from gst-meet via jvb to sender, keeping the SSRC of original stream? I've tried to modify the rewriteRtcp method in AdaptiveVp8Projection with no luck, even if I turned off Rtcp rewriting entirely - it didn't work. Unfortunately it doesn't work also when I select the 180p low layer in gst-meet, where technically the sender SSRC matches the projection's SSRC, so it could be a NACK timing issue?

I've also tried to completely skip vp8 adaptive projection and forward only a single spatial layer to gst. It didn't help, even with 2000ms latency jitterbuffer in gst, the stream didn't end up with continuous sequence numbers. Maybe NACKing 3 simulcast streams is too taxing for a jittered network environment with packet loss?

Steps to reproduce

-enable simulcast
-break the networking

IFACE=<network interface>
TARGET=<public ip of jvb> # conference-dev-2
tc qdisc del dev $IFACE root
tc qdisc add dev $IFACE root handle 1: htb
tc class add dev $IFACE parent 1: classid 1:1 htb rate 300mbit
tc filter add dev $IFACE parent 1: protocol ip prio 1 u32 flowid 1:1 match ip dst $TARGET
tc qdisc add dev $IFACE parent 1:1 handle 10: netem delay 25ms 5ms distribution normal loss 5%

-try to capture the jitsi endpoint with gstreamer, the video in motion will be heavily broken
-when the network is fine, the motion artifacts disappear:
tc qdisc del dev $IFACE root

Environment details

• latest jvb and gst-meet on linux
• trust-bwe false
• layer suspension turned off
• 3 simulcast layers 180p, 360p and 720p (lowering the bitrate to super low values did not help)
• gst-meet pipe
• ./gst-meet --verbose --web-socket-url=wss://srv/xmpp-websocket --last-n=1 --select-endpoints= --xmpp-domain=--focus-jid=focus.meet.jitsi --muc-domain=muc.meet.jitsi --recv-video-height=720 --room-name= --recv-pipeline-participant-template="vp8dec name="video" ! videoconvert ! autovideosink"