Clarifying OpenSSL version
Closed this issue · 7 comments
Hello! I was looking into the OpenSSL version on Heroku 16 and was surprised by the version number. It looked to me like the version was 1.0.2g and hadn't been updated in two years, which would be surprising. So I started looking at the OpenSSL changelog and saw that this was lagging the latest branch (1.1.0h), but even within 1.0.2 it looked like it was over two years behind on patches (latest is 1.0.2o). This is what I see with the Ruby buildpack set to 2.5.1, run on a heroku-16 dyno this morning:
$ openssl version -a
OpenSSL 1.0.2g 1 Mar 2016
built on: reproducible build, date unspecified
platform: debian-amd64
I can see in the Heroku docs on stacks that heroku-16 includes openssl 1.0.2g-1ubuntu4.12 (and that heroku-18 has 1.1.0g-2ubuntu4). When I asked about this over in the Ruby buildpack repo, @schneems helpfully pointed me over here and suggested that this isn't actually 1.0.2g but contains additional security patches. And then I found the Heroku changelog that there have been several security patch notices for Heroku 16 (eg this recent one), although they don't reveal what they are for. While I suspect you all are on top of this, it'd be helpful to learn more about this aspect of security for Heroku deployments.
Can you help me understand more about how this works, and where I can see what code exactly is running for OpenSSL? This might be as simple as helping understand what 1.0.2g-1ubuntu4.12 points to, or how heroku-16.sh installs the package (perhaps it's pulled in from libgnutls-openssl27?)
Thanks in advance for the help in learning more about the process you all use for updating security-related packages. Past explanations like #69 (comment) from @tt and #87 (comment) from @cji have been awesome and helpful for understanding how you all work, and gaining confidence in Heroku 👍
Kevin,
We use the default Ubuntu package for OpenSSL. So 1.0.2g-ubuntu4.12 points to this package: https://launchpad.net/ubuntu/+source/openssl/1.0.2g-1ubuntu4.12
Each stack image is built using Docker and based on the Ubuntu image with the same version on top of which we install additional packages (that's what heroku-16.sh does).
See https://github.com/heroku/stack-images/blob/master/heroku-16/Dockerfile
We then (this part isn't within this repository) extract the docker image's files into our own format to be used as the stack image.
That base is used as the base for all running dynos on that stack. When we update the stack image, running dynos keep using the previous one. But as they get cycled once every day, their next restart uses the new version.
In the case of OpenSSL here, the package is provided by Ubuntu in the base image. They apply security patches directly there.
When we update the stack images for security issues, we post it on our changelog.
The last update was on june 18: https://devcenter.heroku.com/changelog-items/1436
Finally, you can see every installed package on each stack and their version here: https://devcenter.heroku.com/articles/stack-packages
This article is automatically updated whenever we update the stack image on the platform.
Hoping that clarifies things.
@dmathieu Thanks for your help! This is awesome for understanding the pieces of how the dynos are built, and the link to the Ubuntu openssl 1.0.2g-1ubuntu4.12 package is super helpful too.
Can you help me understand why I see this date on a production heroku-16 dyno?
$ openssl version -a
OpenSSL 1.0.2g 1 Mar 2016
built on: reproducible build, date unspecified
platform: debian-amd64
This date doesn't seem to line up with either the release date on the Ubuntu package you referenced, or the June 18th patch in the Heroku changelog. But it does correspond with the release date of the original OpenSSL 1.0.2g. If I understand the pieces, one hypothesis is that the the Ubuntu patches have kept that date fixed at the original release 1.0.2g date, and haven't set the "build date" which the openssl binary says is unspecified.
Separately, as someone who is looking to Heroku as a trusted platform, can you help me understand more about why you all think that the Ubuntu 1.0.2g-1 patch is the best one for keeping Heroku secure? One thing that's hard for me to understand is that the OpenSSL changelog shows several CVEs and patches since the original 1.0.2g, and the Ubuntu 1.0.2g-1 doesn't seem to include all of those.
To be explicit, I don't know enough about the details in all those patches, and I'm looking to learn more about this and not criticizing any practices here. A first step I took towards learning more about the security of our Heroku deployment was to check that we are running the latest version of critical software with no known CVEs, and I was surprised to see this wasn't the case and could use some help understanding. Thanks for your help!
The reason the build date and version don't change is because Canonical applies the correction patch directly to the package.
You can see that in the diff for the latest CVE: http://launchpadlibrarian.net/366381026/openssl_1.0.2g-1ubuntu4.11_1.0.2g-1ubuntu4.12.diff.gz
So the version doesn't change, nor does the release date.
Applying patches instead of using the latest version is the method Canonical uses in order to provide better stability. They guarantee you won't get any breaking changes. Only security fixes.
We apply software updates once every 2 years, as we follow Canonical's LTS release cycle. The heroku-16 stack is Ubuntu 16.04. heroku-18 is Ubuntu 18.04, which was released last april.
So what really matters isn't that we're using 1.0.2g, but 1.0.2g-1ubuntu4.12, which is based from one OpenSSL version and applies all security patches on top of it. All CVEs are indeed applied.
Hoping this is more clear now.
It should also be noted that this is the standard way any system is maintained. The practice is called backporting, and it's how Red Hat, Debian, Ubuntu and so forth operate: for a long-term support release, a freeze is made to specific package versions at a cutoff date, and for the duration of support, security fixes from newer versions are ported to the old version by the maintainers.
The alternative would be to follow upstream package releases. The result would be that after a short while, applications would start crashing left and right: you cannot run a program (e.g. the Ruby interpreter) that was compiled against e.g. OpenSSL 1.0.2 with OpenSSL 1.1.0, because the API and ABI of the two are different.
@dmathieu @dzuelke you both are awesome, thanks for your help in understanding this!
@dmathieu The diff is super helpful! I see some of the CVEs in the openssl changelog listed there (four in the helpful diff you shared), but I also see many others that are not included there. I'm assuming that's because the Canonical and Heroku security team has reviewed these and decided they aren't worth including in patches. To take one concrete example, CVE-2016-2178 in the changelog section titled "Changes between 1.0.2h and 1.0.2i [22 Sep 2016]" sounds to me like it would be serious, that it was included in a OpenSSL patch version after 1.0.2g, but that it hasn't been included in the Canonical patch based on the diff you shared, and therefore that Heroku dynos would be exposed to this. Can you help me understand that particular example? I apologize if I'm misreading or misunderstanding these pieces, and thanks for your help!
@dzuelke Yeah, this is a great explanation and matches my understanding, thanks! I understand why an LTS version would fix on OpenSSL 1.0.2 since 1.1.0 breaks compatibility, and earlier @schneems awesomely pointed out to that Heroku 18 has moved to 1.1.0, which was super helpful! The only remaining bit I'm trying to understand is why Canonical hasn't considered other parts of OpenSSL releases important enough to include them as security patches (eg, CVE-2016-2178 to take a concrete example), and to learn more about why the Heroku security team is confident in this.
I suspect either I'm misunderstanding something, or the Heroku security team has good reasons for taking this approach that I just don't know about. I appreciate you helping me to learn more in depth about how this aspect of security on Heroku works!
That CVE was fixed in Ubuntu: https://people.canonical.com/~ubuntu-security/cve/2016/CVE-2016-2178.html (the exception seems to be 0.9.8 on Trusty, but we're not running that). Generally, Debian, Ubuntu, Red Hat etc backport everything that can be backported, sometimes with considerable effort if the upstream version series is no longer maintained.
@dzuelke Awesome, thanks! So I think what I was mixed up on was that the diff that @dmathieu awesomely shared was diffing openssl_1.0.2g-1ubuntu4.11 with openssl_1.0.2g-1ubuntu4.12 and I misread it as the diff of the latest with base 1.0.2g.
To come full circle, separate from this conversation it looks like https://launchpad.net/ubuntu/+source/openssl/1.0.2g-1ubuntu4.13 was published a few hours ago. So at some point shortly, it'll be incorporated into the next build of the Heroku 16 stack, and then visible in the Heroku changelog and packages list if I'm understanding correctly.
Thanks for all your help and teaching me about how this works and how you all work to keep things on Heroku secure! 👍