configuration option for preferred compression method

[kwinsch]

There seems to be no consensus in what order the supported compression methods shall be in the "Accept-Encoding" request header field. My current Chrome browser is sending "gzip, deflate, br, zstd", which results in both nginx and angie to prefer gzip over zstd for proxied servers, at least where the original content is already delivered in a compressed form from the proxied server. This is leading to a situation where more bandwidth is used by the clients than technically possible.

As a web server administrator , I would like to configure the order of preference on the server side, to enforce and prefer "zstd" if it is included in the clients "Accept-Encoding" field. The same should be possible for the brotli and other supported compression types. The preferred compression method shall be enforced, even when a proxy_pass destination delivers content with an other compression method.

The Content-Encoding would therefore have a similar behavior to the HTTP/2 and HTTP/3 module, where the protocol can be upgraded between the client<->reverse-proxy, and the connection to the back-end is using HTTP/1 or HTTP/1.1.

Example
client<---zstd--->angie<---gzip--->webserver

Tested using Chrome 127 on Linux, queering FreeBSD 14.1 with angie 1.6.0 and angie-module-zstd-1.6.0, using the webserver llc's repository.

[VBart]

Neither gzip module, nor zstd account the "Accept-Encoding" field order. The only reason why you get gzip in the mentioned case is that your backend response is already compressed with gzip.

client<---zstd--->angie<---gzip--->webserver

That just doesn't make any sense. In order to achieve such scenario, Angie will have to decompress gzip and then recompress to zstd, which is very inefficient operation and just waste of resources adding more latency. You will get worse performance with that than just sending gzip as is (current behavior).

Anyway you can do that if you will, but you need to configure the gunzip module for decomprssion of gzip from your proxied server: https://angie.software/en/configuration/modules/http_gunzip/

[kwinsch]

I completely agree with you, that in most cases this is a waste of resources and of course adds delays. I work on a special case, where we serve some pages over a very low bandwidth radio connection with enough compute power on both ends. So saving some bytes over a small latency and compute penalty is more than acceptable.

But it seems, that I am not the only crazy one, that either for testing or for other reasons do something similar, especially if you are not able to change the proxied server software.

1. google/ngx_brotli#162
2. nextcloud/docker#2076

Especially in that case, where someone wants to upgrade old software with brand new features implemented in the proxy, might be of more value than some think.

Thanks for the hint with the http_gunzip. Will try that out and compare the performance with the linked workarounds.

[VBart]

In order to work together with the gunzip module properly, you may need to compile the zstd module from this revision: tokers/zstd-nginx-module@f4ba115

Our packages currently contain the release revision of the module, which doesn't include that particular commit. We will update it with the next release.

[VBart]

But for your case I recommend to use brotly instead, since it allows to achieve better than zstd compression ratios. Also, its module has no such issue with ordering in current version and will work together with gunzip out of the box.

[kwinsch]

Quickly hacked in the required functionality in my conf file. For my controlled use case, everything works fine. Always amazing how flexible and adaptive this piece of software is. Now running a 20 year old application served over HTTP/3 using zstd compression without any modifications.

Leaving here the relevant parts of the config for others.

# Check if zstd is in the Accept-Encoding header
map $http_accept_encoding $has_zstd {
    ~*zstd 1;
    default 0;
}

# Determine the used Accept-Encoding value
map $has_zstd $my_accept_encoding {
    1 "zstd";
    0 $http_accept_encoding;
}

server {
...
    location / {
...
        proxy_set_header Accept-Encoding $my_accept_encoding;

        # Decompress proxy_pass content, if client supports zstd
        gunzip on;

        proxy_pass ...
...

[kwinsch]

Closing.