Compression doesn't happen when express sends a `Buffer`

Question

Compression doesn't happen when express sends a `Buffer`

Spaxe opened this issue 7 years ago · 11 comments

As the title suggests, when I do res.send(Buffer.from(...)), the response is sent without gzip encoding.

I'm using app.use(compression()), and the express app is sending the buffer via a router.

Answer 1 · 2017-08-24T01:57:41.000Z

Hi @Spaxe I made a simple server and res.send(Buffer.from(aReallyLongString)) compressed just fine. Is it possible you can provide the following information?

Version of this module you're using
Version of Node.js you're using
Complete source code that demonstrates the issue
Instructions on how to run the provided code (i.e. the request you made)

Thanks!

Answer 2 · 2017-08-24T01:59:58.000Z

Thanks @dougwilson. Node v8.0.0 and compression 1.7.0.

I'll try to get you a minimal test case as soon as I can.

Answer 3 · 2017-08-24T02:13:30.000Z

Thanks! I tried those versions and it still worked fine. Here is the test code and request I'm using if it helps you:

var express = require('express')
var compression = require('compression')

var app = express()

app.use(compression())

app.get('/', function (req, res) {
  res.type('txt')
  res.send(Buffer.from(
    'this is a string, a really long string that is long enough to get over ' +
    'the default threshold for compression. this is a string, a really long ' +
    'string that is long enough to get over the default threshold for ' +
    'compression. this is a string, a really long string that is long enough ' +
    'to get over the default threshold for compression. this is a string, a ' +
    'really long string that is long enough to get over the default threshold ' +
    'for compression. this is a string, a really long string that is long ' +
    'enough to get over the default threshold for compression. this is a ' +
    'string, a really long string that is long enough to get over the default ' +
    'threshold for compression. this is a string, a really long string that is ' +
    'long enough to get over the default threshold for compression. this is a ' +
    'string, a really long string that is long enough to get over the default ' +
    'threshold for compression. this is a string, a really long string that is ' +
    'long enough to get over the default threshold for compression. this is a ' +
    'string, a really long string that is long enough to get over the default ' +
    'threshold for compression. this is a string, a really long string that is ' +
    'long enough to get over the default threshold for compression.'))
})

app.listen(3000)

$ curl -is --compress http://127.0.0.1:3000/
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/plain; charset=utf-8
ETag: W/"4b9-qlGutIQDiDDWf9V3KmxJ0XhwMkU"
Vary: Accept-Encoding
Content-Encoding: gzip
Date: Thu, 24 Aug 2017 02:12:35 GMT
Connection: keep-alive
Transfer-Encoding: chunked

this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression.

Answer 4 · 2017-08-24T02:14:33.000Z

I'll give it a shot and get back to you!

Answer 5 · 2017-08-24T03:40:54.000Z

@dougwilson Reproduced.

Buffer is binary and it should be Content-Type: application/octet-stream. Note that if you omit the res.type() call, express will by default send it as application/octet-stream anyway.

Note the response does not have Content-Encoding: gzip.

var express = require('express')
var compression = require('compression')

var app = express()

app.use(compression())

app.get('/', function (req, res) {
  res.type('application/octet-stream')
  res.send(Buffer.from(
    'this is a string, a really long string that is long enough to get over ' +
    'the default threshold for compression. this is a string, a really long ' +
    'string that is long enough to get over the default threshold for ' +
    'compression. this is a string, a really long string that is long enough ' +
    'to get over the default threshold for compression. this is a string, a ' +
    'really long string that is long enough to get over the default threshold ' +
    'for compression. this is a string, a really long string that is long ' +
    'enough to get over the default threshold for compression. this is a ' +
    'string, a really long string that is long enough to get over the default ' +
    'threshold for compression. this is a string, a really long string that is ' +
    'long enough to get over the default threshold for compression. this is a ' +
    'string, a really long string that is long enough to get over the default ' +
    'threshold for compression. this is a string, a really long string that is ' +
    'long enough to get over the default threshold for compression. this is a ' +
    'string, a really long string that is long enough to get over the default ' +
    'threshold for compression. this is a string, a really long string that is ' +
    'long enough to get over the default threshold for compression.'))
})

app.listen(3000)

$ curl -is --compress http://127.0.0.1:3000/
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/octet-stream
Content-Length: 1209
ETag: W/"4b9-qlGutIQDiDDWf9V3KmxJ0XhwMkU"
Date: Thu, 24 Aug 2017 03:39:00 GMT
Connection: keep-alive

this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression. this is a string, a really long string that is long enough to get over the default threshold for compression.

Answer 6 · 2017-08-24T03:46:17.000Z

Hi @Spaxe , that is expected; by default this module will not attempt to compress if the Content-Type is not compressible (see https://github.com/expressjs/compression#filter):

filter

A function to decide if the response should be considered for compression. This function is called as filter(req, res) and is expected to return true to consider the response for compression, or false to not compress the response.

The default filter function uses the compressible module to determine if res.getHeader('Content-Type') is compressible.

You can provide your own filter function to use the detection method you like.

Answer 7 · 2017-08-24T03:51:33.000Z

Thanks @dougwilson - I will do that :)

To clarify my use case, I'm sending https://github.com/mapbox/geobuf as binary streams, and it's designed to be compressible.

Answer 8 · 2017-08-24T04:00:43.000Z

Hi @Spaxe yea, just because it's binary doesn't mean it's not compressible :) If you are using a specific content-type for that response, we can get it listed as compressible so it will compress by default instead of you needing to add a custom filter (let me know what the content-type is). Since the compression in Node.js occurs on the thread pool, it can be quite an expensive operation, so the filter is there to try not to compress responses that will not really benefit as a trade-off for poor Node.js compression performance.

Answer 9 · 2017-08-24T04:04:08.000Z

Would it be advisable to use application/octet-stream+text? Is there a downside for doing this?

Answer 10 · 2017-08-24T04:08:42.000Z

So the content-type is really just a signal to whatever your clients are. They may not care, but if they do care what it is, then you would need to use something they would understand and not reject.

If you do use that type, it's marked as compressible out of the box with this module, since the media type as the text suffix. Ideally the suffix should generally be ignored by generic interpretations of the content-type, but if the client (a) doesn't like the + or (b) is doing a comparison of application/octet-type without dropping the suffix it may not work, though it's client-dependent.

I hope that makes sense.

Answer 11 · 2017-08-24T04:09:33.000Z

Hmm. I need to read more on this. Thanks @dougwilson.