How to decode byte array from Websocket?

[VictorZhanUnity]

I build a WebSocket server, it will keep broadcasting every frame of the camera to clients.

I build a client with Unity and receive stream data(frame encode to byte array) from the server correctly.

But I don't know how to decode it with ffmpeg.

I study this project to understand how it works, but still don't know how to decode byte array to imageData.

[radek-k]

What does the byte array contain? Is it H264 stream, MJPEG or something else? What library are you using for encoding the stream on the server side?

[VictorZhanUnity]

What does the byte array contain? Is it H264 stream, MJPEG or something else? What library are you using for encoding the stream on the server side?

Yes, It's H.264 stream, It encoding by my teammate. I dunno which library he used for encoding.

I can receive the byte array sent from the WebSocket server, but just don't know how to decode it and transfer it texture2d.

[Asamaclar]

@VictorZhanUnity hey, i have same problem that you, i have byte arrays compressed in h264 sent from linux device to the server and redirect to my wpf client and i need show frames into my window. I had same architecture without video encoded with h264 and all good less the network traffic. Now i need a decoder library to decode all frames (byte arrays) in .net and i dont know what lib i could use.

How you resolve your problem?

[Asamaclar]

Well, i solved this problem this ffmpeg wrapper < https://www.codeproject.com/Articles/5337264/NET-Wrapper-of-FFmpeg-Libraries > libs and this code:

using FFmpeg;
using System.Drawing;
using System.Runtime.InteropServices;

namespace RemoteDrive.ImageTools
{
    public class ToolsFFmpegLib
    {
        public int fps {  get; set; }
        private AVCodecParserContext parserctx;
        private AVCodecContext decoderctx;
        public ToolsFFmpegLib()
        {
            AVPacket avpack = new FFmpeg.AVPacket();
            decoderctx = new AVCodecContext(AVCodec.FindDecoder(AVCodecID.H264));
            parserctx = new AVCodecParserContext(decoderctx.codec_id); 
        }
        public Bitmap DecodeH264(byte[] messagedata)
        {
            Bitmap bmp = null;
            var lenmsg = messagedata?.Length??0;
            if (decoderctx.Open(null) == 0 && lenmsg>0)
            {
                AVPacket pkt = new AVPacket(lenmsg);
                AVFrame frame = new AVFrame();
                IntPtr unmanagedPointer;
                unmanagedPointer = Marshal.AllocHGlobal(lenmsg);
                Marshal.Copy(messagedata, 0, unmanagedPointer, lenmsg);

                
                int ret = parserctx.Parse(decoderctx, pkt, unmanagedPointer, lenmsg);
                // First time doesnt work, i reppet.
                ret = parserctx.Parse(decoderctx, pkt, unmanagedPointer, lenmsg);

                // Once we have packet data
                if (pkt.size > 0)
                {
                    bool got_frame = false;
                    decoderctx.DecodeVideo(frame, ref got_frame, pkt);
                    if (got_frame)
                    {
                        bmp = AVFrame.ToBitmap(frame);
                        frame.Free();
                        fps++;
                    }
                    pkt.Free();
                }
                // Call unmanaged code
                Marshal.FreeHGlobal(unmanagedPointer);
            }
            return bmp;
        }
    }
}

inside my wpf viewmodel code

 Bitmap bitmap = toolimg.DecodeH264(imagebytes);

                    if (bitmap != null)
                    {
                        var ms = new MemoryStream();
                        bitmap.Save(ms, System.Drawing.Imaging.ImageFormat.Bmp);
                        Debug.WriteLine($"FPS: {toolimg.fps}");

                        bitmapImage.BeginInit();
                        bitmapImage.StreamSource = ms;
                        bitmapImage.CacheOption = BitmapCacheOption.OnLoad;
                        bitmapImage.EndInit();
                    }