[SUGGESTION] Get rid of the digital gain automatic in manual exposure mode
cpixip opened this issue · 12 comments
As far as I know, libcamera handles a user request for a specific exposure time by balancing two camera parameters, namely exposure time and digital gain. I guess the main reason for this approach is that a given hardware setting is able to realize only a discrete set of exposure times in hardware. In order to "supply" the user with the possibility of requesting arbitrary exposure times, libcamera simulates them by choosing a closeby exposure time which is realizable in hardware and adjusts the digital gain so that the product of both equals in a way the exposure time requested by the user.
This approach creates a whole lot of problems in any task faintly related to any scientifc application:
- the combination of exposure time times digital gain libcamera comes up with is not exact enough and varies; images taken one after the other with fixed exposure times vary visually in image brightness. That spoils any project where the camera is used as a measurement device.
- the libcamera algorithm needs sometimes several frames to get close to the requested exposure time. This spoils the fun when requesting several exposures in rapid succession, as in HDR work. Libcamera kind of "cheats" sometimes by presenting the user an image which was not taken by the exposure time requested, but for example with an exposure time half the value requested, and with a digital gain of two for compensation. Of course, such an image will have a noticable larger noise level in dark image areas.
One way around these problems would be to make the digital gain setable again. If this happens, the adapting algorithm described above could simply be deactivated. Activation of the algorithm (normal mode) could happen again by setting the digital gain to zero.
With the adaptive algorithm switched off, the user will usually not get the exact exposure time requested, but he could figure this out easily by looking at the metadata returned for this frame. He could even find out the exactly realizable exposure times by simply requesting (with the automatic running) an exposure time close by and monitor what the camera is acutally delivering in terms of exposure time and digital gain. If the digital gain is close to 1.0 (I currently use a threshold of <1.005), you "discovered" an exposure time directly usable with the hardware you have at hand.
One hope of mine (and others) would be, that in this way rapid changes of exposure times would be again realizable by this approach.
Currently, my software monitors the metadata of returned frames, checks the digital gain and only sends the captured frame to a client when the digital gain has settled to a value close enough to 1.0. Typically, this takes about 4 to 6 frames. Or, in other words: potentially, the capture speed I can currently achieve might be 4 to 6 times faster than with the current approach, while giving more consitent exposures between frames.
You can see the effect of the digital gain adapting algorithm in the first 300 captures in the diagram below:
The measured image brightness jumps between two noticably different values even so it should be more like a solid curve (as at times >300). As the exposure value/digital gain combination not only effects the data output of libcamera, but also the data contained in the raw image data, this behaviour is definitely a show-stopper for many scientific applications.
Hi, thanks for the info an all this, I know we've discussed some of these issues before. There is work going on, but perhaps it's my fault because I've not been keeping folks in the loop (and it's taking a long time). Anyway, perhaps now's the time to post an update.
The first problem we have is with the digital gain varying "unexpectedly". This happens because exposure and analogue gain require "a few frames" to take effect, but digital gain doesn't. When it calculates the digital gain, and sees new exposure and analogue gain values, it doesn't know what it had in mind "a few frames ago" when it set those values.
A patch set to fix this has been in review for quite some time, the latest version was posted a couple of weeks ago. I'm optimistic that these changes will get merged fairly soon now, perhaps within a week or two, but bear in mind that we're not in control of the timescales!
Another hot topic in the libcamera world is "per frame controls". This is the idea that you should be able to tell easily what control values have been set when the application receives a completed request (explicitly comparing the values you think you changed is somewhat awkward). We've made proposals for how we would like to see this work, and even implemented it as a prototype, but I think the timescales for this are significantly longer again.
As part of our "per frame controls" proposals, we can push control changes up the request queue so there'll be less delay before they take effect. And the corresponding version of Picamera2 has functions that take a list of controls lists, applies them all in a pipe-lined fashioned, and spits the completed requests out at you as they arrive.
Hope that's all useful...!
Thank you for this reply. I must confess, I do not really understand the details of all this (but I'd love to). Is there any place where these kind of things are documented? Specifically: the type of algorithm setting the digital gain?
In the good old days when I started working with digital cameras (some 30 years ago), setting a certain exposure time had immediate effect. Because, as far as I understand it, it's just setting a hardware register. Same should be the case for analog gain and digital gain. They are normally realized at the sensor level. So at most, this spoiled (again, in the good old days) a single frame, depending on the sensor and the way the software was synced to the sensor operation.
So why do exposure and analog gain "require 'a few frames' to take effect"? Or are you referring here to the time it takes for certain setting to be propagated through libcamera's command pipeline? But that's not what my proposal is all about.
In most scientific settings, you just would want to specific exposure time, analog gain, digital gain (sufficient for raw processing) as well as red and blue gain, optionally the ccm, for jpg or similar output formats. And keep them fixed for the period of observation as best as you can.
Well, at this point in time, you can set analog gain as well as the white-balancing red and blue gains to the values you need; the ccm you might need can be taken care of by an appropriate tuning file. So there is a way to switch off most automatic procedures of libcamera in some way or other.
My suggestion (switch off the digital gain "magic" when digital gain is set to a user value, similar in behavior to the whitebalance algorithm) is targeting the only remaining automatic algorithm which you can not control as a user. And this "automatic" is nasty, as it affects even raw image data.
Just to clarify: I am not at all concerned by the delayed response of the libcamera command pipeline. That is a thing one can live with, especially as the frames are delivered with appropriate metadata. One can simply wait until the frame with the appropriate metadata arrives. But some automatic algorithm modifying even the values of the raw images makes it hard to use the Raspberry Pi hardware for a lot of interesting science projects.
Thank you for this reply. I must confess, I do not really understand the details of all this (but I'd love to). Is there any place where these kind of things are documented? Specifically: the type of algorithm setting the digital gain?
Documentation... that's difficult. In my experience an awful lot of this stuff is proprietary and secret, and the companies involved never publish any of it. Our Raspberry Pi Libcamera Tuning Guide is quite a departure in this respect, and all our source code is fully open, which I have never seen before.
The digital calculation is simply "the total effective exposure I wanted n frames ago divided by current exposure time * analogue gain (which are the ones I wrote to the sensor n frames ago)" (or will be, once our patches are merged).
In the good old days when I started working with digital cameras (some 30 years ago), setting a certain exposure time had immediate effect. Because, as far as I understand it, it's just a setting a hardware register. Same should be the case for analog gain and digital gain. They are normally realized at the sensor level. So at most, this spoiled (again, in the good old days) a single frame, depending on the sensor and the way the software was synced to the sensor operation.
So why do exposure and analog gain "require 'a few frames' to take effect"? Or are you referring here to the time it takes for certain setting to be propagated through libcamera's command pipeline? But that's not what my proposal is all about.
Both. There are several frames of delay just getting values through the software pipeline and libcamera, though as I said, we're expecting to be able to reduce those.
But there are intrinsic frame delays in the sensor too. These are worse because they're rolling shutter sensors, so by the time you read out pixels for one frame, the next frame is already being exposed.
All in all, the "exposure delay" in the sensor is nearly always 2 frames. Analogue gain is often only 1 frame of delay, though sometimes the sensor makes it the same as the exposure because it thinks you'll like that, and I've seen vertical blanking delays up to 3 frames.
In most scientific settings, you just would want to specific exposure time, analog gain, digital gain (sufficient for raw processing) as well as red and blue gain, optionally the ccm, for jpg or similar output formats. And keep them fixed for the period of observation as best as you can.
The only values that affect raw frames are the exposure time and analogue gain. Everything else is applied by the ISP - and they simply cannot have any affect on the raw data. Once you see a particular pair of (exposure and analogue gain) values in the image metadata, those images are 100% repeatable.
Well, at this point in time, you can set analog gain as well as the white-balancing red and blue gains to the values you need; the ccm you might need can be taken care of by an appropriate tuning file. So there is a way to switch off most automatic procedures of libcamera in some way or other.
Digital gain, colour gains, CCM, LSC are all applied by the ISP. It should be possible to disable them all completely either through libcamera controls, or the tuning file when libcamera doesn't have them. (Though the digital gain, as noted previously, still requires those patches.)
My suggestion (switch of the digital gain "magic" when digital gain is set to a user value, similar in behavior to the whitebalance algorithm) is targeting the only remaining automatic algorithm which you can not control as a user. And this "automatic" is nasty, as it affects even raw image data.
Just to clarify: I am not at all concerned by the delayed response of the libcamera command pipeline. That is a thing one can live with, especially as the frames are delivered with appropriate metadata. One can simply wait until the frame with the appropriate metadata arrives. But some automatic algorithm modifying even the values of the raw images makes it hard to use the Raspberry Pi hardware for a lot of interesting science projects.
Please do check the behaviour of raw files w.r.t. exposure time and analogue gain (and digital gain having no effect). We report these metadata values correctly 100% of the time.
(There is a sensor/driver bug in the imx477 which can stop the exposure time going up as much as you would expect, but maybe ask me about that when you see it.)
Thank you very much for this detailed reply!
To summarize:
- raw image captures are only affected by exposure time and analog gain settings. Great!
- digital gain is applied in the ISP pipeline and affects the final output of libcamera.
Judging from your comment above (italics mine):
The digital calculation is simply "the total effective exposure I wanted
nframes ago divided by current exposure time * analogue gain (which are the ones I wrote to the sensornframes ago)" (or will be, once our patches are merged).
So it seems that there is currently a syncing problem between exposure/analog gain and digital gain? That would explain the behavior I am seeing in the output of the libcamera stack. So there is at least hope that the brightness variations one sees currently will be reduced in the near future?
With the old ISP it was possible to set the digital gain to a constant value. This way, image intensities were rock solid and reproducible over time. Honestly, in the libcamera context, I don't understand the rationale for replacing this simple approach with a free-running automatic algorithm that has no way to turn it off.
I would therefore like to speak out again in favor of making all automatic functions that libcamera offers to be able to be switched off if necessary. From my understanding, the digital gain automatic is the only one left which can not be circumvented.
If the "possibility to set the digital gain to a fixed value" proposal would be implemented, the full manual control of the camera will significantly expand the possible uses of the Raspberry Pi hardware in the field of science and education.
Hi again, to answer the various questions:
...
- raw image captures are only affected by exposure time and analog gain settings. Great!
- digital gain is applied in the ISP pipeline and affects the final output of libcamera.
Yes to both.
...
So it seems that there is currently a syncing problem between exposure/analog gain and digital gain? That would explain the behavior I am seeing in the output of the libcamera stack. So there is at least hope that the brightness variations one sees currently will be reduced in the near future?
Yes to both again. The brightness variation won't be reduced, it will be gone (with that one minor proviso about the imx477).
With the old ISP it was possible to set the digital gain to a constant value. This way, image intensities were rock solid and reproducible over time. Honestly, in the libcamera context, I don't understand the rationale for replacing this simple approach with a free-running automatic algorithm that has no way to turn it off.
The idea that there's a "simple approach" is actually an illusion. As soon as we let people set the digital gain, the following week we'll get the bug "I set exposure X analogue gain Y and digital gain Z but they don't happen at the same time". To make all this work we have to account for all the variable frame delays which is where all the complexity lies, and if the legacy stack is managing synchronise all this correctly then it isn't "simply" setting the digital gain. This is exactly what our in-progress patch set does, and we're working hard to get them merged!!
...
David,
many thanks for your effort! This here
The brightness variation won't be reduced, it will be gone (with that one minor proviso about the imx477).
will be a great step forward. Let's hope that these patches (probably those?) get merged soon!
Can you elaborate a little bit more on the details of the "minor proviso" with respect to the IMX477? Is that related to the little spikes I am observing here:
These are traces of the average image intensity with exposure times 250000/62486/31243/15625. The shorter exposure times are fine. The metadata of the frames is not mirroring these tiny spikes, digital gain stays always at 1.000 units.
In closing, while a number of users would certainly appreciate the full ability to manually control a camera within an open-source context, I can understand that this would probably result in a flood of support requests as to why different settings take effect at different times. So I get your point.
Still, I'd love to see an open-source camera without any hidden automatism! I think a lot of people wouldn't mind at all if the different adjustments were done asynchronously. In a scientific context, reproducibility is important, but not whether the device takes a short time to adjust to changing settings.
@cpixip, the patches for the digital gain "bug" have now been merged onto the libcamera tree. We are in the process of packaging an apt release with the latest code (likely ready next week), but if you are able to build libcamera libraries yourself, you can git it a go straight away.
Good news - that will really help with HDR-work!
Just one comment to add. There is a bug with the imx477 (or its driver) that we've never got to the bottom of and it can result in exposure values "going wrong" in a particular way (and will also have a knock-on effect on the gain). It means that for this to work correctly, we need to run the sensor at a fixed framerate (but long enough for your longest exposure). For example call
picam2.set_controls({"FrameDurationLimits": (33333, 33333)})
for 30fps . If you can, call this before starting the camera otherwise this too will have several frames of latency before taking effect. To the best of our knowledge, other sensors are not affected.
Thanks David for this additional comment! Would it be possible to elaborate a little bit in more detail what going wrong in a particular way (and will also have a knock-on effect on the gain) specifically means? If I request a certain exposure time/analog gain combination, will I get that, but the reported metadata will be wrong? Or is the reported metadata as requested, but the image did use in fact some different values?
One further question, just to understand your framerate workaround better. My usecase takes several different exposures of a scene. Will the workaround help if I set a range for the framerate? I.e., not using a single fixed value (as in your example above), but a lower value different from the higher one?
Thanks David for this additional comment! Would it be possible to elaborate a little bit in more detail what going wrong in a particular way (and will also have a knock-on effect on the gain) specifically means? If I request a certain exposure time/analog gain combination, will I get that, but the reported metadata will be wrong? Or is the reported metadata as requested, but the image did use in fact some different values?
The problem relates to changing the the frame period (or framerate). When using variable framerates, you may find that the frame period is shorter than the requested exposure time, so the frame period (also referred to as the frame length, or vertical blanking) must be increased. A strange property of the imx477 is that the frame period sometimes (but not always) changes a frame later than we expect. The result is that the frame with that longer exposure will actually get clamped by the frame length that hasn't changed.
For example, if you're running at variable framerates and have been requesting 10ms exposure, and let's support that that camera can't go faster than 50fps (20ms). Now you ask for 30ms exposure.
The increase in vertical blanking will be a frame late, so the first frame that should have 30ms exposure will be stuck at 20ms. Only on the next frame will it go up to 30ms. As a result of the iincorrect exposure, that in-between frame will end up with a digital gain of 1.5, because it wants to make it look like the 30ms you asked for.
Note that the returned metadata is always accurate, whether this goes wrong or not.
One further question, just to understand your framerate workaround better. My usecase takes several different exposures of a scene. Will the workaround help if I set a range for the framerate? I.e., not using a single fixed value (as in your example above), but a lower value different from the higher one?
No, not in general. If the vertical blanking is changing at all then you risk this happening. It all depends on your exposure times, of course. In full res mode you can't have a frame length less than 100ms. so it's not an issue if you're always asking for < 100ms exposure.
Thank you for the clarification. Great that the metadata is always accurate - this allows to easy check the returned frame. I use anyway a previous recipe from you, where I wait until the right combination of exposure time and digital gain is delivered. Great!