H264 packet in AVC mode needs to create AVCDecoderConfigurationRecord

[KallDrexx]

When H264Packet is created with is_avc: true, all payloads that are depacketized are in the AVC format with the length prepending each nalu. This is correct except for SPS and PPS nalus, as AVC encoding expects these to be configured in an AVCDecoderConfigurationRecord.

Is this something you'd be open to a PR for to have that built into H264Packet?

The code I currently use for this is:

pub fn convert_to_avc_decoder_config_record(
    sps_records: &Vec<Bytes>,
    pps_records: &Vec<Bytes>,
) -> Option<Bytes> {
    if sps_records.is_empty() || pps_records.is_empty() {
        return None;
    }

    let mut bytes = BytesMut::new();
    bytes.put_u8(1); // version
    bytes.put_u8(sps_records[0][1]); // profile
    bytes.put_u8(sps_records[0][2]); // compatibility
    bytes.put_u8(sps_records[0][3]); // level
    bytes.put_u8(0xFC | 3); // reserved (6 bits), nalu length size - 1 (2 bits) ????
    bytes.put_u8(0xE0 | (sps_records.len() as u8)); // reserved (3 bits), num of SPS (5 bits)
    for sps in sps_records {
        bytes.put_u16(sps.len() as u16);
        bytes.extend_from_slice(&sps);
    }

    bytes.put_u8(pps_records.len() as u8);
    for pps in pps_records {
        bytes.put_u16(pps.len() as u16);
        bytes.extend_from_slice(&pps);
    }

    Some(bytes.freeze())
}

At first glance it seems like a path to implementing this is for the H264Packet type to store an Option<Bytes> field for both a cached sps and cached pps. When it caches both a sps and pps then that depacketize() call would return the AVCDecoderConfigurationRecord bytes. Calls to depacketize() calls before receiving the sps and pps records would always return an empty payload.

I'm not totally clear how this gets handled when new SPS and PPS records come down from RTP, especially since AVC based systems I'm working with (RTMP) seem to ignore later sequence headers, so I'm not sure if H264Packet needs to contain a collection of seen SPS and PPS headers, or only the first ones.

[k0nserv]

We have migrated this crate to the monorepo(webrtc-rs/webrtc) please re-open this issue over there.