Stream bitmap ( single frame image ) to the virtual device

Question

Stream bitmap ( single frame image ) to the virtual device

hesa2020 opened this issue 2 years ago · 4 comments

hesa2020 commented 2 years ago

Hello I looked around and tried a few things.
if I run manually the command it works:

"C:\ffmpeg\bin\ffmpeg.exe" -loop 1 -framerate 1 -i temp.jpg -r 1 -pix_fmt rgb24 -f rawvideo - | "C:\Program Files\AkVirtualCamera\x64\AkVCamManager.exe" stream MyFakeCamera0 RGB24 640 480

In my c# app this is what I do:

cameraProcess= new Process();
ProcessStartInfo startInfo = new System.Diagnostics.ProcessStartInfo();
startInfo.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
startInfo.FileName = "C:\\Program Files\\AkVirtualCamera\\x64\\AkVCamManager.exe";
startInfo.Arguments = "stream MyFakeCamera0 RGB24 640 480";
startInfo.RedirectStandardError = true;
startInfo.RedirectStandardInput = true;
startInfo.RedirectStandardOutput = true;
startInfo.UseShellExecute = false;
startInfo.CreateNoWindow = true;
cameraProcess.StartInfo = startInfo;
cameraProcess.Start();

And then to send my image to the camera:

Task.Factory.StartNew(async () =>
{
    while(true)
    {
        try
        {
            MemoryStream outputStream = new MemoryStream();
            await FFMpegArguments
            .FromFileInput(Path.Combine(Environment.CurrentDirectory, "temp.jpg"))
            .OutputToPipe(new StreamPipeSink(outputStream), options => options
                .WithFrameOutputCount(1)
                .WithFramerate(1)
                .ForcePixelFormat("rgb24")
                .ForceFormat("rawvideo"))
            .ProcessAsynchronously();
            var data = outputStream.ToArray();
            cameraProcess.StandardInput.BaseStream.Write(data, 0, data.Length);
        }
        catch(Exception ex)
        {
            Console.WriteLine(ex.Message);
        }
        await Task.Delay(1000 / 30);
    }
});

However I really dislike the idea of starting 33 process per second to stream an image.

Then I see my temp.jpg being shown as the virtual camera.

What would be the easiest way to get my bitmap as rgb24 format ( I have no clue how this format actually works ) and write it to the device. ( without using ffmpeg )

Answer 1 · 2022-10-02T18:12:57.000Z

I don't use C#, but if I understand well, you are trying to mimic the command line arguments, right?

If you want to keep that way, what you must do is open both processes, connect the stdout of FFmpeg to the stding of the manager (which I understand you are doing here), and then configure FFmpeg to the equivalent parameters of -loop 1 -framerate 1, so is FFmpeg who will keep looping over and over, sending the frames to the virtual camera, and you won't need the while(true) loop, so that way you will have one unique process for FFmpeg and one unique process for AkVCamManager. What i understand from your code is that you are launching the process every single loop.

Now, the best solution is no using FFmpeg, but instead using the C# API for reading the JPG, PNG, or whatever format (which I'm pretty sure C# has one, here is a possible solution), read the raw pixel data from the image, convert it to 24 bits RGB if necessary, then open the AkVCamManager, and keep sending the raw frames to the stdin in a loop, maybe using a clock or a sleep. So that way you just need one process and no more dependencies than the virtual camera. You can try adapting the C example to C#.

I have no clue how this format actually works

Every raw image can be read into an array of data, were each pixel is (commonly) read in occidental writing direction, from left to right, from up to down. Each pixel is composed of a number of components, were the number of components and it's type vary from format to format, in the case of RGB, it has 3 components: Red, Green, and Blue, also every component has a depth which is the number of bits needed to represent the component, if for example I say 24 bits RGB, that means that the component R has 8 bits, the component G has 8 bits, and the component B has 8 bits (3 * 8 = 24), so:

Reading the 1st byte of the array will give you the red component of the top left corner,
Reading the 2nd byte of the array will give you the green component of the top left corner,
Reading the 3rd byte of the array will give you the blue component of the top left corner,
Reading the 4th byte of the array will give you the red component of the 2nd pixel on the same line and in the right of the previous one,
Reading the 5th byte of the array will give you the green component of the 2nd pixel on the same line and in the right of the previous one,
Reading the 6th byte of the array will give you the blue component of the 2nd pixel on the same line and in the right of the previous one,
and so on...

| R0G0B0 | R1G1B1 | R2G2B2 | R3G3B3 | ...

The size in bytes of each line of the image (also called stride) can be calculated as:

lineSize = bpp * width

bpp means Bytes Per Pixel which in the case of 24 bits RGB is 3 bytes (3 components of 1 byte each one).

the total size of the image in bytes is :

size = lineSize * height

For example an image of 24 bits RGB 640x480, the line size would be 1920 bytes, and the total size would be 921600 bytes, and I say "would be" because I don't want to complicate the theory.
In 32 bits RGB format (bpp = 4), the first component would be unused then you read RGB.
In 32 bits ARGB format (bpp = 4), the first component is the alpha component (for controlling the transparency of the pixel) then you read RGB.
RGB24, RGB32, and ARGB are the more commonly used pixel formats for RGB, and those will be the ones the C# API will gives you.

Also, in Windows, images are commonly mirrored vertically, so the first line you read is the bottom, and the last line is the top.

Answer 2 · 2022-10-03T00:03:39.000Z

Hello thank you the function "GetRawBytes" from one of your link is what I needed.
Its working great but something weird is happening.
I am reading a jpg file that is supposed to be Bit depth 24.
I am creating a new Bitmap graphic and drawing my jpg onto it and then I convert that to raw bytes and stream it.
However the image has a blue tone to it as if it is missing a color channel.

Lets say it was RGB32 or ARGB32 instead of RGB24, wouldnt it be missing the Blue channel ?

Edit:
Figured the GetRawBytes was wrong.
Here is how I got it to work correctly:

public static byte[] GetRawBytes(Bitmap bmp)
{
    if (bmp == null) throw new ArgumentNullException("bmp");
    if (bmp.PixelFormat != PixelFormat.Format24bppRgb) throw new InvalidOperationException("Image format not supported.");
    BitmapData data;
    byte[] raw;
    //Lock the bitmap so we can access the raw pixel data
    data = bmp.LockBits(new Rectangle(0, 0, bmp.Width, bmp.Height), ImageLockMode.ReadOnly, bmp.PixelFormat);

    //Accessing pointers must be in an unsafe context
    unsafe
    {
        int w = bmp.Width,
            h = bmp.Height,
            bmpStride = data.Stride,
            rawStride = 3 * w;
        byte* bmpPtr = (byte*)data.Scan0.ToPointer();
        //Allocate the raw byte buffer
        raw = new byte[3 * w * h];
        fixed (byte* rawPtr = raw)
        {
            //Scan through the image and copy each pixel
            for (int r = 0; r < h; ++r)
            {
                for (int c = 0; c < rawStride; c += 3)
                {
                    rawPtr[r * rawStride + c] = bmpPtr[r * bmpStride + c + 2];//red
                    rawPtr[r * rawStride + c + 1] = bmpPtr[r * bmpStride + c + 1];//green
                    rawPtr[r * rawStride + c + 2] = bmpPtr[r * bmpStride + c];//blue
                }
            }
        }
    }
    bmp.UnlockBits(data);
    return raw;
}

Answer 3 · 2022-10-04T14:47:20.000Z

Good! sometimes can happen that the components red and blue are swapped, it depends on how the platform and the libraries define the format, if the arrangement depends on the byte endianness, if the stride is aligned or not, and many other things.

if (bmp.PixelFormat != PixelFormat.Format24bppRgb) throw new InvalidOperationException("Image format not supported.");

What you could do next is, instead of rejecting anything but Format24bppRgb, read the documentation to see which other formats are supported, and implement a proper reading/conversion function for that format.

for (int r = 0; r < h; ++r)
{
    for (int c = 0; c < rawStride; c += 3)
    {
        rawPtr[r * rawStride + c] = bmpPtr[r * bmpStride + c + 2];//red
        rawPtr[r * rawStride + c + 1] = bmpPtr[r * bmpStride + c + 1];//green
        rawPtr[r * rawStride + c + 2] = bmpPtr[r * bmpStride + c];//blue
    }
}

I would rather change that for this:

for (int y = 0; y < h; y++) {
    int offsetY = y * stride;

    int offsetRi = offsetY + 2;
    int offsetGi = offsetY + 1;
    int offsetBi = offsetY;

    int offsetRo = offsetY;
    int offsetGo = offsetY + 1;
    int offsetBo = offsetY + 2;

    for (int x = 0; x < rawStride; x += 3) {
        rawPtr[offsetRo + x] = bmpPtr[offsetRi + x]; // Red
        rawPtr[offsetGo + x] = bmpPtr[offsetGi + x]; // Green
        rawPtr[offsetBo + x] = bmpPtr[offsetBi + x]; // Blue
    }
}

The inner loop should always do just the bare minimum number of operations for speed, in graphics processing, the more operations you do, the bigger the penalty you get in the frame rate.

Answer 4 · 2022-10-07T01:55:26.000Z

int offsetRi = offsetY + 2;
int offsetGi = offsetY + 1;
int offsetBi = offsetY;

int offsetRo = offsetY;
int offsetGo = offsetY + 1;
int offsetBo = offsetY + 2;

I've recently discovered that having to swap red and blue components is a bug in the virtual camera 😕, it is not a big thing, nothing to worry about, but I will have to fix that later.