Contributing microcode versions to MCE.db
Closed this issue · 8 comments
My name is Richard Hughes and I run the LVFS which is a firmware site used by most of the consumer OEMs to distribute mostly UEFI firmware to Linux users. We've recently switched from chipsec uefi decode to UEFIExtract to decompress FV volumes from capsules. This gives us microcode blobs "for free" which we're currently saving as blobs in a private database.
I was wondering two things:
-
If we can use MCE.db to tell vendors "you're shipping an old microcode version" when they upload a firmware update with microcode updates for the same CPUID.
-
If in return, we can contribute new microcode versions automatically/semi-automatically as they are uploaded somehow. Ideas I had here were perhaps having a script that emailed you (?) a tarball of all the microcode versions that existed on the LVFS every week, or perhaps hitting an endpoint you provide (e.g. MCE_upload.py with POST data). Ideas very welcome here.
a tarball of all the microcode versions
Of course, I'm assuming you want the microcode blobs themselves. It would be trivial to generate a:
INSERT INTO Intel VALUES('000A0654','00000022','000000C2','20191113','00016400','2D6E88E1');
INSERT INTO Intel VALUES('000A0650','00000022','000000BE','20191010','00016400','B5D77A0C');
as a "secret" page on the LVFS...
@hughsie Although I'm not related to @platomav - creator of these useful repos and utilities - I'd like to thank you for your excellent ideas: they could improve the availability of new microcodes to the community and also reduce the amount of UEFIs containing the outdated microcodes (which the users are forced to manually patch into their UEFI sometimes - not trivial and error prone process).
Yes, you could use MCE.db, as I'm not aware of any confusing situations (especially for the modern CPUs) where the incompatible CPUs had the same CPUID. Regarding the microcode submission, if the automatic approach won't be possible (better to ask @platomav about this!), this could be done manually i.e. once per 1-2 months as they're accumulating - would take just a few minutes of your time. Will be nice if you could check if your already collected set contains anything newer than https://github.com/platomav/CPUMicrocodes and please share with us by a pull request to this repository after forking it.
please share with us by a pull request to this repository
My concern with doing this was that's unreviewable, i.e. it's just a binary blob that gets slight larger. You can't be sure what I've taken out, added, etc. I'm guessing @platomav would want the actual microcode binary too.
If we can use MCE.db to tell vendors "you're shipping an old microcode version" when they upload a firmware update with microcode updates for the same CPUID.
Of course. This is what MCE does for its end-users at the last row (Last Yes/No). The logic for that can be found at "mc_upd_chk_intel" and "mc_upd_chk_amd" functions. Something to keep in mind: when it comes to microcodes (firmware in general I guess), latest is not always better or tested. Unlike us, manufacturers have some insider/confidential info on what got changed/fixed at newer microcode releases so if they ship older microcodes, it's not always due to boredom or indifference. It could be that newer versions have not been thoroughly tested, have been retracted/downgraded by Intel or not contain anything important enough to warrant a new BIOS version. To sum up, such a message ("you're shipping an old microcode version") would be good to show but won't necessarily be a bad thing.
If in return, we can contribute new microcode versions automatically/semi-automatically as they are uploaded somehow.
Sure. You can either use MCE with the latest MCE.db for that and check for any "was not found at the database, please report it" message or simply use MCE.db directly with the aforementioned logic in mind to determine yourself if a microcode is found at the database or not. For MCE, any microcode which is reported as missing from the DB is welcome, even from 1995.
Ideas I had here were perhaps having a script that emailed you (?) a tarball of all the microcode versions that existed on the LVFS every week
That could work. LVFS can determine what's not in the MCE DB and then send these via an email or similar.
I'm guessing @platomav would want the actual microcode binary too.
Of course. Simply adding entries to MCE.db does not guarantee their authenticity and it does not help further tool development and binary understanding. For example, research such as Intel Microcode Extra Undocumented Header would not have been possible without the binary database that we've been collecting for years.
To sum up, such a message ("you're shipping an old microcode version") would be good to show but won't necessarily be a bad thing.
I was thinking something like:
Simply adding entries to MCE.db does not guarantee their authenticity
So you'd like me to upload them to https://www.win-raid.com/t3355f47-Intel-AMD-VIA-amp-Freescale-CPU-Microcode-Repositories-Discussion-25.html manually?
I was thinking something like
Ah yes, that messaging looks good.
So you'd like me to upload them to Win-Raid manually?
Right, this would also work if you don't mind posting manually.
It could be that newer versions have not been thoroughly tested, have been retracted/downgraded by Intel
Here's a real time example of microcode updates wrecking havoc on some recent CPUs: intel/Intel-Linux-Processor-Microcode-Data-Files#24
Right, this would also work if you don't mind posting manually.
Although it's the status quo to have them on that forum I would think there are cleaner ways to do this, like maybe make another repo that LVFS automatically uploads new microcodes in stable with the output/metadata as commit message. Then it's much easier for a job to pick up and import all that. Another job can also verify authenticity of metadata too.
The way I see it, LVFS is already open for review. Anyone can download an uploaded firmware image and get the microcodes themselves if they desire. Sending me an email of new binaries through automation will be helpful to MCE and make its maintenance easier. Also, a lot of times OEMs tend to include some PRE (PRD Test or ES/QS) microcodes at their BIOS images which I do not include at CPUMicrocodes but are very important to have for further MCE development and microcode research/understanding. Having a fully transparent repo with some PRE microcodes might be...uncomfortable for Intel so sending them via email helps us while also leaving them open for review in case someone wants to manually download a given BIOS and extract them via MCE or similar.
By the way, when it comes to microcodes, you don't have to worry about authenticity because the CPU Loader will not load any non proper/official microcodes. It's all taken care of by Intel via digital signatures. Unfortunately we cannot verify the RSA Signature as far as I can tell, but MCE checks everything else.