Add support for Akvcam
fangfufu opened this issue · 47 comments
Chrome doesn't like v4l2loopback, but Akvcam is known to work correctly with Chrome. An example script has been provided to interface with Akvcam using Python.
I have no problem with chromium (did you use the exclusive_caps option for v4l2loopback). I wonder if there is really a difference.
But I am open for more ideas, as long as they do not complicate the program too much. As scheduling the frames currently is only a small part of the code, it could probably made modular, so you can change the backend (I am looking if there is a Windows/Mac backend anyway).
The examples in https://github.com/webcamoid/akvcam/wiki/Usage-and-examples look the same as the command lines for v4l2loopback. Maybe it can just be used as drop-in replacement for v4l2loopback?
I think v4l2loopback expects RGB, and akvcam expects YUV2. I definitely have problem with Chrome - I haven't tried Chromium yet.
The replacement should be fairly easy, I will do that to my own Linux-Fake-Background-Webcam, when I have time. (It is still being maintained - people have been sending me pull requests)
I wonder if this can be solved by converting the RGB numpy array to YUV and keep using pyfakewebcam.
chrome and chromium should behave the same, except for some Google integration, the updater and the PDF reader.
I also tried with electron apps, but that's chromium, too.
pyfakewebcam does exactly this, see https://github.com/jremmons/pyfakewebcam/blob/master/pyfakewebcam/pyfakewebcam.py
What framerate do you achieve? I think chrome only recognizes cams that provide a certain framerate like 5 fps or so.
This may be a cause for problems with chrome. And this could possibly avoided by duplicating frames, this would need some experimentation. Especially how the frame rate is measured, so if you are allowed to send 5 frames at once and then process the next one for a second or if you need a thread to send the duplicates with some time in between.
I think v4l2loopback (no idea about akvcam) can configure a framerate, maybe this leads to buffering the frames that are too fast.
I think I achieve about 8fps or something, there is a thread about why Chrome doesn't work with v4l2loopback. (umlaeute/v4l2loopback#78)
Interesting thread and linked issues, but I see no clear resolution either. Some posts sound like it could be an issue with the driver version (I am using debian testing with version 0.12.5).
Did you test akvcam and it works reliable for you? I did not test it, yet.
Akvcam worked fine for me. There seems to be some instability issue, but it works mostly fine.
Do you want to open an issue in the pyfakewebcam project? Maybe more users can profit from pyfakewebcam supporting akvcam as well.
Hmm, I am a bit busy right now. Feel free to open an issue. I reckon it would be easier to just modify the code and submit a pull request.
I first want to test akvcam myself. I got the kernel module working, now I need to experiment with client code.
@allo- , FWIW, I was chatting to some random people on my old university's IRC channel, they all complained about v4l2loopback not working with Ubuntu properly... So I think I will try and integrate Akvcam to my scripts by the end of this week.
I did only try it on debian testing yet. When your code snippet is working, I can probably integrate it.
@drjerk1 should probably add a license to this post or create a small project for it, though.
@allo- you can intergrate it if you want, but i copy pasted some parts of it from other github sources
@drjerk1 Are there some projects that can already be used?
I am still thinking if it should be integrated into pyfakewebcam or a fork of it.
@allo- I wrote it for my university project, which can swap faces in the video based on single photo, like deepfake do but with less quality, which i will publish in some time
@drjerk1 This sounds like an interesting project.
But with my question I hoped that you may know some module where parts of the code are from, that can be used.
But maybe your code can be built into a simple module like the pyfakewebcam module. I need some more time to experiment with it before I know how I would like to use it.
@allo- sadly i've no experience working with virtual cameras before, but i think that it can write to any fake webcam driver because linux files like /dev/video1 provides universal api themselfs (with close, open, ioctl syscalls), it just writes bytes to file and opens it properly
Just for the record, I added akvcam support on my implementation, please feel free to steal my code. I think the main thing is the configuration file in /etc/akvcam, it needs to have rw support. Akvcam is pretty much a drop-in replacement for v4l2loopback.
I'ill have a look.
@fangfufu you seem to have a typo here: https://github.com/fangfufu/Linux-Fake-Background-Webcam/blob/master/fakecam/akvcam.py#L57
When you tested using this code, you'll probably have swapped colors without the typo.
Nope, that's intended. OpenCV reads stuff in BGR format.
If you look at https://github.com/fangfufu/Linux-Fake-Background-Webcam/blob/master/fakecam/fake.py, and look for akvcam in the source code, you would see that the changes I made are minimal.
Are you sure that the unused variable iamge is intended? You have typo in the variable name and because of this the converted frame is never used.
When the frame looks good nevertheless, you can probably just remove the line and the frame does not need to be converted at this place.
I need to test it in detail later. I just noticed the typo there and that your code is not using the converted frame.
Ah yes, you are right, there is a typo. I definitely could not read... I tested the class itself with fake.py it works really well. Anyways, the colour is a bit off when I run the example script... I suppose it is just that I hadn't run the example script for a while.
I need more time to test such stuff. I forked pyfakewebcam after the author added a note that it's unmaintained. Maybe it would be useful to merge it there. But your class looks like it can be used as drop-in replacement, so I can just add it to the project and load the right module for the driver the user is using.
Well I think somebody needs to maintain pyfakewebcam. But adding my class is pretty much minimal effort. It doesn't really matter which way you go about it - I just think akvcam is superior than v4l2loopback.
After reading in the pyfakewebcam source that looks very similar to your code, I found out that you can just configure akvcam to use YUYV2 as format.
Adapted from https://github.com/webcamoid/akvcam/wiki/Configure-the-cameras I just removed the RGB24 format as input format:
This config works without any modifications of my code:
[Cameras]
cameras/size = 2
cameras/1/type = output
cameras/1/mode = mmap, userptr, rw
cameras/1/description = Virtual Camera (output device)
cameras/1/formats = 2
cameras/2/type = capture
cameras/2/mode = mmap, rw
cameras/2/description = Virtual Camera
cameras/2/formats = 1, 2
[Formats]
formats/size = 2
formats/1/format = YUY2
formats/1/width = 640
formats/1/height = 480
formats/1/fps = 30
formats/2/format = YUY2
formats/2/width = 640
formats/2/height = 480
formats/2/fps = 20/1, 15/2
[Connections]
connections/size = 1
connections/1/connection = 1:2
You are literally contradicting what hipersayanX said. Are you sure it works?
Okay it seems other people have reported similar experience as you.
I worked a few minutes ago when I tested it ;-).
Maybe the akvcam docs are a bit outdated?
I feel hipersayanX isn't exactly sure what his code does. ;) He literally told me that the output device only takes RGB24 as input...
I am sticking with my implementation anyway, it does fewer colour space conversion.
@allo- , look: webcamoid/akvcam#26 (comment)
I do not know why it works, but it works. And the example config in the Wiki has RGB24, YUY2 what looks like it is offering both (and preferring RGB24?).
I think I rather document how to configure akvcam, so I do not need to duplicate code that is almost the same in pyfakewebcam.
I may reconsider implementing it directly, when the frame conversion becomes the bottleneck.
The code in webcamoid/akvcam#26 (comment) looks VERY similar to pyfakewebcam.py and both seem to do the color conversion using img_yuv = cv2.cvtColor(image, cv2.COLOR_RGB2YUV).
Pyfakewebcam has in addition a numpy-only fallback, but the rest looks similar.
I am not absolutely sure about all the reshape/concatenate operations in the two codes, but I guess they should have the same result in the end.
The reshape and concatenate are done because of this:
https://www.linuxtv.org/downloads/legacy/video4linux/API/V4L2_API/spec/re09.html
I am going to stick to my code and let akvcam to handle all the nonsense. My super long config file was actually generated by webcamoid. I want to config to be compatible with webcamoid as well.
I suppose the reshaping in your code and the copying into the buffer in the pyfakewebcam code are equivalent. But I am not too motivated to verify if it is really the same as long as the resulting image is the same.
Good to know that webcamoid can generate configs. I will see what I'll add for documentation.
I'll come back at your code, if there are problems with my approach or reasons why it should be handled in another way.
I think the rest of our data formats keep compatible (RGB / BGR frames) anyway and the schedule frame part can be swapped out for something else at any time.
When the processing is too slow, the standby frame is flashing when using akvcam, while v4l2loopback just is slowed down (as it is just a loopback).
@fangfufu Does your implementation solve this problem?
Ohh, I encountered the same problem as well! No, I haven't managed to solve it! It happened to me only once.
I guess there is some minimum frame rate, which may be configurable?
I have absolutely no idea...
Actually you can set the frame rates here:
https://github.com/webcamoid/akvcam/wiki/Configure-the-cameras
Considering we are just writing stuff to akvcam, if we stop writing, it is reasonable for akvcam to revert to the default screen.
Okay, I am adopting your config file to support Akvcam on my end.
So what do you think do we need for this issue? Maybe just adding a akvcam example config file?
I have always shipped a config file. I have adjusted my config file to look very similar to yours.
https://github.com/fangfufu/Linux-Fake-Background-Webcam/blob/master/akvcam/config.ini
But yes, with this config file, you can use pyfakewebcam to output directly to Akvcam.