ngx.location.capture + proxy_pass == stuck, no response

[egonSchiele]

I have two locations like this:

location ~* ^/proxy_test/(.*?)/(.*) {
    proxy_pass_request_headers off;
    proxy_pass http://$1/$2;
}

location ~* ^/test/(.*?)/(.*) {
    set $a $1;
    set $b $2;
    content_by_lua '
    local url = "/proxy_test/" .. ngx.var.a .. "/" .. ngx.var.b
    res = ngx.location.capture(url)
    ngx.say(res.status)
  ';
}

Then I make a request to test/static.adit.io/guitar.jpg. This should:

1. make a subrequest to /proxy_test/static.adit.io/guitar.jpg, which will download the image and serve it
2. the response gets saved to res
3. the status gets printed out.

However, this request never finishes. Turning debugging on, here's what I see:

The subrequest gets made:

2014/08/22 23:56:33 [debug] 36366#0: *4 http proxy header:
"GET /guitar.jpg HTTP/1.0
Host: static.adit.io
Connection: close

A response is received:

2014/08/22 23:56:33 [debug] 36366#0: *4 recv: fd:39 214 of 4096
2014/08/22 23:56:33 [debug] 36366#0: *4 http proxy status 200 "200 OK"
2014/08/22 23:56:33 [debug] 36366#0: *4 http proxy header: "Server: nginx/1.2.3"
2014/08/22 23:56:33 [debug] 36366#0: *4 http proxy header: "Date: Fri, 22 Aug 2014 23:56:33 GMT"
2014/08/22 23:56:33 [debug] 36366#0: *4 http proxy header: "Content-Type: image/jpeg"
2014/08/22 23:56:33 [debug] 36366#0: *4 http proxy header: "Content-Length: 641458"
2014/08/22 23:56:33 [debug] 36366#0: *4 http proxy header: "Last-Modified: Wed, 02 Jul 2014 04:30:27 GMT"
2014/08/22 23:56:33 [debug] 36366#0: *4 http proxy header: "Connection: close"
2014/08/22 23:56:33 [debug] 36366#0: *4 http proxy header: "Accept-Ranges: bytes"
2014/08/22 23:56:33 [debug] 36366#0: *4 http proxy header done

Then I see plenty of this (not sure what to make of it). The last lines in the log are:

2014/08/22 23:56:34 [debug] 36366#0: *4 pipe buf in   s:1 t:1 f:0 00000000019132C0, pos 00000000019132C0, size: 32768 file: 0, size: 0
2014/08/22 23:56:34 [debug] 36366#0: *4 pipe length: 375432

But there has been no response from the server. My request just hangs for minutes until I finally kill it.

Lua version: LuaJIT 2.0.3 -- Copyright (C) 2005-2014 Mike Pall. http://luajit.org/
Nginx version: nginx/1.2.9
CentOS release 6.5 x86_64

[egonSchiele]

Full config: https://gist.github.com/egonSchiele/bddde74bb68be92126df

[egonSchiele]

The image size is 600+ KB. I figured out that setting these two:

proxy_max_temp_file_size 0;
proxy_buffers 8 32k;

was causing the issue. If I enable temp files, or increase the limit of the buffers, everything works as expected. However, I would expect the server to return an error if the buffer size was too small, instead of hanging forever.

[agentzh]

@egonSchiele Thank you for the report! I can reproduce the issue you're seeing on my side with the following minimal and self-contained example:

location = /main {
    content_by_lua '
        local res = ngx.location.capture("/sub")
        ngx.say(res.status)
        ngx.say(#res.body)
    ';
}

location = /sub {
    proxy_redirect off;
    proxy_pass_request_headers off;

    proxy_max_temp_file_size 0;
    proxy_busy_buffers_size 4k;
    proxy_buffers 2 4k;

    proxy_pass http://127.0.0.1:$server_port/backend;
}

location = /backend {  # emulate a backend emitting large data
    content_by_lua '
        ngx.print(string.rep("a", 16*1024))
    ';
}

Accessing /main hangs the request as you've described. I'll investigate the issue shortly and get back to you soon :)

[agentzh]

@egonSchiele Okay, I've located the issue to be in ngx_event_pipe (used by ngx_http_upstream, and finally ngx_http_proxy_module). It directly checks the condition "downstream->data != p->output_ctx" which does not make sense in ngx_lua's modified version of the subrequest model. There is no easy fix without patching the nginx core.

In your specific use case, because you're using ngx.location.capture to buffer the whole response in memory anyway, you can increase your proxy buffers setting, for example.

It's highly recommended to use one of those lua-resty-http libraries based on ngx_lua's cosocket API, for example,

https://github.com/pintsized/lua-resty-http

The nginx subrequest thing is a dead end at least for ngx_lua and it's too painful to push its implementation any further.

[DracoBlue]

I wrote about this issue at https://dracoblue.net/dev/hanging-subrequests-in-nginx/ and fixed it by setting proxy_max_temp_file back to something bigger then 0 (e.g. 1024m).

The other solution was to set proxy_buffering off.

Is there any way to workaround this and keep the caching?

[agentzh]

@DracoBlue I've already made it clear in my previous comment in this issue that the problem is caused by proxy_max_temp_file 0 and there is no easy fix for that without patching the nginx core. Just avoid disabling buffering by using a bigger value, as you've already found out.

[agentzh]

@DracoBlue proxy_cache is surely supported in the subrequest. It is non-buffered streaming processing that causes the problem.

[DracoBlue]

@agentzh thanks!