Some questions about merging with latest jsmpeg version
roelvz opened this issue · 5 comments
Hello,
Do you think it's feasible to merge the latest version of jsmpeg into jsmpeg-vnc? I would like to do this because the latest jsmpeg version includes an audio decoder. I first tried with video only, but it doesn't work out of the box. The decoder thinks the received steam is garbage. I guess this is because the c++ part does not send a valid MPEG1-stream (because of a custom header each frame).
I tried to remove the custom header, but this does not seem to be working. Do you think it should be that simple? Or is there anything else I could be missing?
A better solution would be to take the custom header into account in the newest version of jsmpeg. Would this be hard? Could you give some directions on where to start?
For the audio part in the c++ code, I think this is pretty straightforward, but any tips are welcome.
Thank you
For jsmpeg-vnc to work with the current jsmpeg version, it needs to send out an MPEGTS Stream. FFMpeg has a muxer to create this stream. So, yeah, it's certainly possible.
I want to come back to this project eventually, but I'm buried with "real" work at the moment. Don't hold your breath :/
Just for info, I wrote c++ code (started from jsmpeg-vnc for the idea but it turned out to me completely new code) that works together with the latest version of jsmpeg and also includes audio.
@roelvz Is your version (with sound) published anywhere? I would be interested to check it out!
Just for info, I wrote c++ code (started from jsmpeg-vnc for the idea but it turned out to me completely new code) that works together with the latest version of jsmpeg-vnc and also includes audio.
Hi Roel....by any chance you could share this?
I cannot make the code available, but here are some tips that may be helpful:
- Instead of only encoding the video (and audio), after the encoding you'll also have to use a muxer to output an MPEGTS stream.
- Encoding in ffmpeg is done using
libavcodec
and is already done in jsmpeg-vnc. Encoding audio is similar. You'll need to useAV_CODEC_ID_MPEG1VIDEO
for video andAV_CODEC_ID_MP2
for audio with 2 channels. This is what I did back then for the audio:
_audioCodec = avcodec_find_encoder(AV_CODEC_ID_MP2);
_audioEncoder = avcodec_alloc_context3(_audioCodec);
_audioEncoder->sample_rate = outputSampleRate;
_audioEncoder->sample_fmt = AV_SAMPLE_FMT_S16;
_audioEncoder->channel_layout = av_get_default_channel_layout(2);
_audioEncoder->channels = av_get_channel_layout_nb_channels(_audioEncoder->channel_layout);
_audioEncoder->bit_rate = audioBitrate;
_audioEncoder->time_base = AVRational{1, _audioEncoder->sample_rate};
auto result = avcodec_open2(_audioEncoder, _audioCodec, nullptr);
- Once you have the encoded video and audio, you need to mux them. Ffmpeg uses
libavformat
for this. Useful functions for initialisation areavformat_alloc_context
,av_guess_format
,avformat_new_stream
,avcodec_parameters_from_context
andavformat_write_header
. For getting the data onto the WebSocket you can useavio_alloc_context
. - The actual encoding happens with
avcodec_send_frame
andavcodec_receive_packet
for both video and audio. - The actual muxing happens with
av_write_frame
for both video and audio. - You may have to resample the audio first. Check out
swr_alloc
(init) andswr_convert
(real time). This happens before encoding off course (comparable to the scaling for video, which is already present in jsmpeg-vnc).
Note: I'm not using this code anymore and it was all build to work with ffmpeg 4.0.2. Some of these functions are probably deprecated or deleted in recent ffmpeg versions.