mrv2 - ProRes4444 incorrect color on Windows and Linux
Closed this issue · 28 comments
I am hoping someone can investigate or provide any help. After installation of this version I am comparing a color checker rendered out of after effects in both AE and mrv2 looking at 32bit and 8bit RGB values. They are off by a large enough factor of concern and it causes video footage to look different enough to cause issues. Here are two screenshots of the difference in values. The color is set to input raw and output raw in mrv2 and preserveRGB with no working space in AE (which is the best way to check color in general), so there should be no discrepancy. I think that the underlying ffpmeg interpretation could be off and would like to figure out how to fix it.
mrv2 64 bits - v1.0.5 - Built Feb 18 2024
A similar issue seen with .mp4 files as well.
Hi. Thank you for reporting the issues.
First things first. Are you on Windows or on macOS?
mrv2 v1.0.5 revamped its RGB to YUV FFmpeg settings based on the work of ASWF, as seen here (but I may have screwed it up):
https://academysoftwarefoundation.github.io/EncodingGuidelines/
https://academysoftwarefoundation.github.io/EncodingGuidelines/ColorPreservation.html#yuv
https://academysoftwarefoundation.github.io/EncodingGuidelines/tests/chip-chart-yuvconvert/compare.html
I mainly tested with the different YUV formats and bitdepths with the h264 codec on Linux as shown in that page.
Regarding Prores in particular, this is what the ASWF currently recomends and, it seems, if you are on Windows or Linux you are limited to 10 bits:
https://academysoftwarefoundation.github.io/EncodingGuidelines/EncodeProres.html
macOS encodes with its proprietary videotoolbox_prores hardware codec, which may not have its limitation, but it might need to use different settings (not sure).
To carry out the tests, save and post (or upload a .zip somewhere) with:
- a 16-bit PNG of your color chart.
- a video in Prores4444 of one frame of that color chart from AE.
- a video in Prores4444 of one frame of that color chart from mrv2.
and install a full version of ffmpeg. If you are on macOS:
brew install ffmpeg
If you are on Windows, download and install ffmpeg binaries from:
https://www.gyan.dev/ffmpeg/builds/#release-builds
With that info, we are ready to begin some tests. This is likely going to be a long thread, as we try things out.
Ok. I found out what it was. The ASWF page is missing some information regarding the ProRes. I did a bunch of testing with all codecs and now they all return a very close conversion. I am compiling a beta that you can get from sourceforge in an hour, compile from source or wait till v1.0.6 is released probably next week.
Check out the just released v1.0.6 of mrv2 on github releases or sourceforge. It has fixes to Prores, VP9, macOS hardware codecs, and much more!
I checked out the new release! I'm getting the exact same values as I did prior in .5 after installing .6 so I will run through the 16bit test png you outlined and post that along with re-installing ffmpeg along with using a fresh machine that potentially has less junk if mrv2 is pulling some dependencies from the system like an old ffmpeg.
Forget about installing ffmpeg. We don't need it. I created a test suite now to test it easily.
Just answer two things:
- What platform are you on? Windows, macOS or Linux?
- Provide me with the color chart you are using.
Windows 10
Here's a zip of the files. Used 1.0 colors and bars for simplicity vs the ACES Rec709 conversion referenced in initial report. Red should be only red in that channel, white should be 1:1:1, ect.
Mainly concerned with the tiff imported into mrv2 showing 1.0 colors vs the ProRes exported from AE not showing them as pure 1.0. The mrv2 export included in the zip is a "PC" color and Rec709 primaries to attempt to get the full 1.0 RGB values, I assume PC vs TV is "limited vs full" in implementation. Just as a test to see what it is doing, not surprised as it's almost impossible to get a proper prores export out of ffmpeg that matches a quicktime from a supported app.
Let me know if there's anything else I can look at.
First, download the current betas of mrv2 v1.0.7 at:
.exe:
https://mega.nz/file/zbpXWSqT#ntBpqSzzYlDr6o7cEmhRBIi24lMqvFMSqmMdWziKYVg
.zip:
https://mega.nz/file/iWY1ALaA#KbufPaHxN9OPWe00PuBuvFH4xcUpgO_gC2JZ0o20m50
mrv2 can create a matching Prores 4444 movie file to a 16-bit image but on macOS. I am attaching here a .zip file with your .tiff of color bars converted to match:
Curiously enough, albeit FFmpeg can read the file (in mrv2 v1.0.7 only), it cannot create it.
mrv2 can create a very close match of a Prores 4444 movie file to an 8-bit image on any platform if you save in YUV_444P_12LE.
You can also create a matching movie file to your 16-bit image if you change codec to VPX. If you then select GBRP_12 you should be in business (mrv2 can load and save those files). But AE might not support VPX.
Thank you again for looking into this. I just want to clarify, the issue is on Windows 10 the .mov file does not have a value of 1:1:1 for white, 1:0:0 for red, 0:1:0 for green, and 0:0:1 for blue like the tiff. Checked again in the .7 release that you linked. I linked the mrv2 .mov just to see how it's handling things.
I've messed with ffmpeg flags for a few days and gave up trying to get 1:1 with something like an Adobe or OSX approved .mov export on windows. So I really appreciate the effort because i was not able to get it even perceptually looking the same. Most workflows rely on Prores review at some point unfortunately.
I could potentially create a lut to balance the minor ffmpeg offset as a big work around worst case.
edit: attached whitepoint screenshot for clarity
I've messed with ffmpeg flags for a few days and gave up trying to get 1:1 with something like an Adobe or OSX approved .mov export on windows.
But does the Adobe .mov file match? The file you sent me did not. I was puzzled by it because they actually license macOS's Prores converter for Windows (according to the .mov metadata). Either:
a) Apple does not make the Windows version equally as good as their macOS one.
b) Adobe's AE does not know how to use their library.
c) Something else?
It does. In Windows AE the .mov labeled AE2023 matches the 16bit tiff to a factor of .0001 (give or take that last value) in 32bit
Most importantly white is 1:1:1
In 8bit they match 100% other than some compression artifacts from the conversion.
make sure you are preserving RGB for any testing purposes. AE's color management can be a nightmare to navigate.
According to Bing's copilot:
Yes, After Effects does support VP8 and VP9 video coding formats through the WebM plugin[(https://www.webmproject.org/tools/). The plugin uses the libvpx library, which is the reference software implementation for the VP8 and VP9 video coding formats.
You can use the WebM plugin to import and export .webm files in After Effects. The plugin is available for free and can be downloaded from the official website.
Does this mean that it's not possible to have correct color interpretation of ProRes in mrv2 because of the ffmpeg libraries being different?
Yes. The ffmpeg codec (prores_ks) only saves 10bits ProRes files even though ffprobe reports it as 12 bits, while the macOS and their Windows licensed codec really save up to 12bits.
I am looking into alternatives. Currently I have two:
-
VP9 (libvpx-vp9) saving with YUV_444P_12LE (yuv444p12le in ffmpeg command line). It matches but won't give you an alpha channel. In order to bring it to AE, you need to compile the libvpx plugin for it, which is freely available. This is something you may do now if you know how to compile the plugin yourself.
-
GoPro Cineform (cfhd codec in ffmpeg). Also to be implemented in v1.0.7. This is the most promising one so far. It is free of licenses and patents. It can save as BGRP_12LE and BGRAP_12LE (with an alpha). It comes with ffmpeg for free as both decoder and encoder. The only downside is that if your editing program does not support ffmpeg's multiple formats properly, it may give you RGBA_U8 or a black frame.
Basically, every codec and company get you with something. They all want money!!!
excellent info! I will keep an eye out for any of your potential solutions. ProRes4444 without an alpha is probably worth looking into for this workflow of utilizing mrv2 as a dailies solution.
Okay, go to:
https://sourceforge.net/projects/mrv2/files/beta/
And download v1.0.7 beta. Try saving a movie with GoPro Cineform with pixel format GBRP_12 or GBRAP_12. Verify it looks okay in mrv2.
Then try importing it into AE. Hopefully AE has implemented ffmpeg correctly (cross fingers!).
According to Bing's copilot, they have!:
GoPro CineForm is a powerful codec that has played a significant role in video production and editing. Let’s explore the software packages that support it:
Adobe Premiere Pro CC, Adobe After Effects CC, and Adobe Media Encoder CC natively decode and encode QuickTime files using the GoPro CineForm codec on both Mac OS X and Windows systems(https://helpx.adobe.com/premiere-pro/using/gopro-cineform-codec.html). This makes it an excellent choice for video editing within the Adobe Creative Cloud suite.
If you’re preparing source video files for editing on Windows/PC, it’s recommended to convert your files into the GoPro/CineForm file format. This format was specifically designed for editing and is compatible with most popular non-linear editing applications, including iMovie, Final Cut Pro, Premiere, AVID, and Vegas(https://gopro.com/en/us/info/static/how-to-prepare-source-video-files-for-editing-windows-pc).
Additionally, GoPro has open-sourced the CineForm codec to help third-party software developers accelerate high-resolution 360° and future workflows. It’s included with Adobe Creative Cloud products, including Adobe Premiere Pro, for both Mac and Windows versions(https://gopro.com/en/us/news/gopro-open-sources-the-cineform-codec)(https://havecamerawilltravel.com/gopro/gopro-cineform-format/).
I have a solution to the color shifts in ProRes4444 on Linux and Windows. It will be there in v1.0.9. The beta at sourceforge has finished compiling for Windows and macOS ARM64.
This 1.0.9 is working way better! The RGB 8bit values are off by like 2 give or take which is more than acceptable.
Can you outline what you changed? Is it possible to to describe in a simple command format like
OutputArgs=-y -xerror -vcodec prores_ks -profile:v {} -vendor ap10 -pix_fmt {} -vtag ap4h -r {}".format(profileV, pixfmt, frameRate))
I want to mention the color in 1.0.9 being slightly off (similar to prior) over NDI if that is something to look into along similar lines as the digging you've done to get the .mov's working as well as they do in this newest version.
Can you outline what you changed? Is it possible to to describe in a simple command format like
You can see the FFmpeg command-line flags on Sam Richards web page (who thanks to him and my own digging were able to get to the bottom of it). NOTE The web page will only show the movies correctly on macOS Safari as it has native ProRes support which no other browser, AFAIK, supports.
https://encode.taurich.org/EncodingGuidelines/tests/chip-chart-yuvconvert/compare.html
I want to mention the color in 1.0.9 being slightly off (similar to prior) over NDI if that is something to look into along similar lines as the digging you've done to get the .mov's working as well as they do in this newest version.
Oh. Wow! Someone actually uses NDI. I am glad it works for you guys! Regarding a color shift, you'll have to ask Adobe and Newtek (well, the new company that now owns NDI) what color space and other info they are using. mrv2 shows images assuming a rec709 display (not taking into account the color modifications of OCIO of course).
Now that you nailed the .mov color within a crazy good (and free) margin of error, I'd love to explore it as an all in one tool!
The color is pretty 1:1 from AE-->NDI Studio monitor; but has a slight over-saturation & levels difference within mrv2. I've been playing with some OCIO configs and selections to see how close I can get it with no luck yet. 1.2, 1.3 (v2 with a custom Display ODT from the 1.2 config), ect... Full vs Limited videos levels... that sort of thing
I'll make another ticket if I have any reproducible info and breadcrumbs. But let me know if you have any other dependencies I could adjust/try out. I might reach out to Newtek like you mentioned to see what they are automagically doing on the back end without any settings adjusted to get so close to Adobe's playback.
edit:
If it helps,
Black levels from 20:20:20 RGB in AE translate to ~ 33:32:31 in mrv2 over NDI. I might just make a lut to preform a basic correction since I'm not sure how to adjust how the color is interpreted under the hood.
Figured they might be related to the .mov ffmpeg but I can move info like this into a new thread if you'd like to keep this one focused to any .mov stuff moving forward.
Sorry for the previous noise. I finally have a fix for the NDI color problem, which, as you suspected was indeed a misues of FFmpeg. Please give it a try to this v1.1.0 beta and report back:
.exe:
https://mega.nz/file/WawFzDJK#8PEQimgTYgL3aUiy9wAkPo6khPVDea_-UXHivRMoshU
.zip:
https://mega.nz/file/OPBVmQRZ#vttllX-1LuIufnt5D9C7Ikm0bK6maNlFkT4pBqBB1h8
I'll respond here since you have noticed it might be a related issue and I'm happy to say that things are much closer. There are some minor differences that are concerning for fully reliable color playback. It's certainly within the realm of "good enough" depending on the content, but I wanted to relay the information and files that could help adjust anything further. When viewing grey content, it is noticeably warmer. I included a screenshot of 1.9 for comparison of when it was looking off.
It seems like mostly the blue (white balance) is what is still just subtly off if there's a way to look into that on a granular level.
Attached is the ACES ap0 exr that I took to a Rec709 ouput using the 1.2 config, an 8bit png export of a few colors, and another with the grey/white values. The differences between AE & mrv2 over NDI are as follows and were compared with the color picker in NDI and an 8bit AE project without any conversion or working space.
AE
144:35:42 Red
202:179:46 Yellow
12:133:57 Green
169:59:128 Magenta
mrv2 NDI
142:35:38 Red
200:179:43 Yellow
12:131:56 Green
167:58:124 Magenta
Grey Values
AE
204:204:201
180:181:180
145:147:147
NDI
204:204:197
179:181:178
144:146:143
edit: I should mention that after opening NDI Monitor and sending the signal through there, the color looks similarly off (which is surprising). I also wish there was a way to fix the RGB-->YUV banding that is increased by lack of bit depth for the free IP version without hardware.
I think something is broken on your methodology of comparison.
To compare NDI output, I used the official NDI Tools from Newtek which include a "Test Patterns" (an NDI test exporting) program and "NDI Studio Monitor" (their official NDI viewing application).
-
I started "Test Patterns" and selected Rec709.
-
I opened "NDI Studio Monitor", connected to the Test Patterns stream, took it full screen and took a screen capture with "Print Screen" for an 8-bit png.
-
I opened the same stream on mrv2 and did the same thing (F12 for presentation mode) and "Print Screen" to capture an 8-bit png.
I am attaching the color bars of both images:
NDI_and_mrv2.zip
I closed the NDI output and panel. Then I took the .pngs into mrv2 and did a wipe between them. They were (are) identical. There are other patterns you can test with in "Test Patterns".
Maybe I should have explained better. NDI and mrv2 both seem to display color 1:1 now (per your test and prior fix: thank you). But both potentially have a white-point that is slightly off. I'm assuming it has to do with a rgb to yuv conversion. I just wanted to capture those differences in values in case there was anything on your end to look at to actually make the implementation better than NDI.
For example the blue value of the left most grey box png attached being 201 in after effects 8bit and 197 over NDI in mrv2. There is a slight color shift between the asset and what is broadcast. Depending on the content color, it is very noticeable between soruce and NDI. The highly saturated bars and test patterns are not.
Gamma being wildly off prior to your fix in mrv2 was the larger concern. And like you mentioned, if NDI's yuv conversion just has some inherent whitepoint issue that is very small, then that could be something to just have to live with.
Gamma being wildly off prior to your fix in mrv2 was the larger concern. And like you mentioned, if NDI's yuv conversion just has some inherent whitepoint issue that is very small, then that could be something to just have to live with.
I am afraid that indeed that will be the case. The NDI 5 SDK supposedly supports 16-bit YUV and direct BGR(A) 8-bit streams too (according to their specs), which would help alleviate the problems you see.
However, when I tested them I got:
- Incorrect colors in 16-bit YUV. Blacks were colored, as if the color leaked onto them or some planes were off. Could not determine if it was an NDI (more likely) or FFmpeg bug (not likely).
- Crashes on RGB(A) tests, as if memory was not initialized.
I coded support for those streams, but it seems the NDI 5 SDK programs don't properly support it.
All fixed and addressed as best as it can be in v1.1.0.