[Cannot Fix] [SRT]: Server hang when publishing with QSV/OBS encoder

[HolgerHuo]

Describe the bug

srt_server {
    enabled on;
    listen 10080;
}

vhost __defaultVhost__ {
    srt {
        enabled on;
        srt_to_rtmp on;
    }
}

When SRT to RTMP is on and incoming streams isn't very protocol compliant ( e.g., QSV H264 on OBS, the produced stream may not contain SPS and IDR on first few packets, also VAAPI/QSV HEVC on OBS), SRS will refuse to process those packets with logs complaining SPS or PPS empty. This might not be an issue but SRS will also hang for other requests. From my observation, SRS will eat up 3 cores (AMD EPYC 7453) and all other requests (SRT/RTMP playback hangs, HTTP requests hang and delay)

Version
SRS/6.0.177(Hang) in Docker

To Reproduce
Steps to reproduce the behavior:

1. Setup SRT2RTMP
2. Publish stream using QSV encoder on OBS
3. Check SRS CPU Usage
4. Test HTTP connectivity to SRS HTTP Endpoints

Expected behavior
Server should return error to those non-compliant streams and not block other requests, or, when I setup an SRT listener using FFPlay (ffprobe -mode listener -v debug -show_packets -i srt://:12345 ), although FFMPEG also complains for no sps on first few packets, the stream is also playable after a few seconds. Maybe srs should also try to handle these cases, at least not hang and block other requests.

Screenshots

Additional context
From my testing, QSV H264/H265 on OBS, VA-API H265 on OBS has these issues. QSV/VA-API on FFMPEG doesn't. My guess is that OBS may be lagging due to some reasons and not producing expected results, I'm working on fix this bug in OBS or also open an issue their side.

[winlinvip]

Root Cause Analysis

When OBS with QSV/VAAPI encoder sends SRT streams without SPS/PPS in the initial packets:

1. The connection does NOT terminate - The error from check_sps_pps_change() is caught and ignored in srs_app_srt_source.cpp
2. No video frames are forwarded - Without SPS/PPS, the video sequence header cannot be generated, so on_h264_frame() is never called
3. Players receive no video - RTMP/WebRTC players connect successfully but see only black screen (audio may work if present)

Why This Cannot Be Fixed in SRS

• H.264/H.265 streams must have SPS/PPS before video frames per specification
• Without SPS/PPS, SRS cannot create valid RTMP/FLV/WebRTC video packets
• Players cannot initialize video decoders without codec parameters (SPS/PPS)
• This is an encoder bug, not an SRS issue

Solution

Fix the encoder:

• Use FFmpeg with QSV/VAAPI instead of OBS (as you confirmed, this works)
• Configure OBS to ensure SPS/PPS is included in keyframes
• Report this to the OBS project as an encoder bug

FFmpeg works because it handles the stream differently, but the stream itself is malformed and violates H.264 specification.

[HolgerHuo]

I have already reported this issue to OBS but I don't think this issue should be concluded here, because this may result in a DOS since bad actors can send suspicious streams to cause the whole server to hang.

Also, although the incoming stream is malformed, maybe SRS should try to handle it just like FFMpeg because in real life there might be such cases when incoming stream breaks midway.