ImageMagick default resource limits are surprisingly low
agrobbin opened this issue · 5 comments
Hi Heroku!
We've been seeing the occasional width or height exceeds limit exceptions in our Rails application coming from the convert ImageMagick utility (some PDFs with long pages being converted to PNGs), and while investigating, I came across Heroku's default resource limits:
The resource limits (width/height in particular) surprised me, and I'm curious if there's any specific reasoning for them. The original PR that introduced this default policy.xml was #119, and unfortunately there isn't a ton of context in there.
Any insight would be appreciated! I didn't want to dramatically increase these without understanding some of the reasoning behind these Heroku/system defaults.
Hi!
To see the differences between the custom policy used in the setup scripts for the stack images and the default Ubuntu policy, I used:
$ docker run --rm -it heroku/heroku:18
root@03e2b77edbff:/# apt-get update
...
root@03e2b77edbff:/# apt-get install --reinstall -o Dpkg::Options::="--force-confask" imagemagick-6-common
...
<Press "D">
It seems that the width/height and other size-based restrictions are the same as the Ubuntu defaults:
(Note that the diff is reversed due to the way dpkg's diff works, the "new" file is the original Ubuntu version)
--- /etc/ImageMagick-6/policy.xml 2020-10-05 16:49:10.000000000 +0000
+++ /etc/ImageMagick-6/policy.xml.dpkg-new 2019-11-11 18:01:05.000000000 +0000
@@ -1,13 +1,80 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE policymap [
+<!ELEMENT policymap (policy)+>
+<!ELEMENT policy (#PCDATA)>
+<!ATTLIST policy domain (delegate|coder|filter|path|resource) #IMPLIED>
+<!ATTLIST policy name CDATA #IMPLIED>
+<!ATTLIST policy rights CDATA #IMPLIED>
+<!ATTLIST policy pattern CDATA #IMPLIED>
+<!ATTLIST policy value CDATA #IMPLIED>
+]>
+<!--
+ Configure ImageMagick policies.
+
+ Domains include system, delegate, coder, filter, path, or resource.
+
+ Rights include none, read, write, and execute. Use | to combine them,
+ for example: "read | write" to permit read from, or write to, a path.
+
+ Use a glob expression as a pattern.
+
+ Suppose we do not want users to process MPEG video images:
+
+ <policy domain="delegate" rights="none" pattern="mpeg:decode" />
+
+ Here we do not want users reading images from HTTP:
+
+ <policy domain="coder" rights="none" pattern="HTTP" />
+
+ Lets prevent users from executing any image filters:
+
+ <policy domain="filter" rights="none" pattern="*" />
+
+ The /repository file system is restricted to read only. We use a glob
+ expression to match all paths that start with /repository:
+
+ <policy domain="path" rights="read" pattern="/repository/*" />
+
+ Let's prevent possible exploits by removing the right to use indirect reads.
+
+ <policy domain="path" rights="none" pattern="@*" />
+
+ Any large image is cached to disk rather than memory:
+
+ <policy domain="resource" name="area" value="1GB"/>
+
+ Define arguments for the memory, map, area, width, height, and disk resources
+ with SI prefixes (.e.g 100MB). In addition, resource policies are maximums
+ for each instance of ImageMagick (e.g. policy memory limit 1GB, -limit 2GB
+ exceeds policy maximum so memory limit is 1GB).
+-->
<policymap>
+ <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
<policy domain="resource" name="memory" value="256MiB"/>
<policy domain="resource" name="map" value="512MiB"/>
<policy domain="resource" name="width" value="16KP"/>
<policy domain="resource" name="height" value="16KP"/>
<policy domain="resource" name="area" value="128MB"/>
<policy domain="resource" name="disk" value="1GiB"/>
+ <!-- <policy domain="resource" name="file" value="768"/> -->
+ <!-- <policy domain="resource" name="thread" value="4"/> -->
+ <!-- <policy domain="resource" name="throttle" value="0"/> -->
+ <!-- <policy domain="resource" name="time" value="3600"/> -->
+ <!-- <policy domain="system" name="precision" value="6"/> -->
+ <!-- not needed due to the need to use explicitly by mvg: -->
+ <!-- <policy domain="delegate" rights="none" pattern="MVG" /> -->
+ <!-- use curl -->
<policy domain="delegate" rights="none" pattern="URL" />
<policy domain="delegate" rights="none" pattern="HTTPS" />
<policy domain="delegate" rights="none" pattern="HTTP" />
+ <!-- in order to avoid to get image with password text -->
<policy domain="path" rights="none" pattern="@*"/>
<policy domain="cache" name="shared-secret" value="passphrase" stealth="true"/>
+ <!-- disable ghostscript format types -->
+ <policy domain="coder" rights="none" pattern="PS" />
+ <policy domain="coder" rights="none" pattern="PS2" />
+ <policy domain="coder" rights="none" pattern="PS3" />
+ <policy domain="coder" rights="none" pattern="EPS" />
+ <policy domain="coder" rights="none" pattern="PDF" />
+ <policy domain="coder" rights="none" pattern="XPS" />
</policymap>Thanks for following up here @edmorley! That's a little surprising, but helpful nonetheless. As images move to higher and higher resolutions, the 16K-pixel dimension limit has started biting us more and more.
I went down the road of adjusting the explicit width/height restrictions for our Heroku app specifically, and things seem to be working as expected, but am curious if the dimension limits should be loosened in general.
Thanks again!
@agrobbin: You can customize the policy by putting a .config/ImageMagick/policy.xml or a .magick/policy.xml in your app. ImageMagick will pick that up.
You can also put other files like magic.xml or delegates.xml in there.
It had to be a little more complicated than that @dzuelke, because of the way ImageMagick caps the configured value based on the highest-priority configuration. So to supersede the defaults, I actually had to configure MAGICK_CONFIGURE_PATH, since the values I wanted to use were higher than the defaults. If I wanted them lower, I could have put it in .config/ImageMagick and everything would've Just Worked!
Closing since we're matching upstream here, and there are ways to customise as mentioned above :-)