worker process exited on signal 11

Question

worker process exited on signal 11

Opened this issue 9 years ago · 12 comments

GoogleCodeExporter commented 9 years ago

What steps will reproduce the problem?
1. using the module in 1.1.19

What is the expected output? What do you see instead?
worker process exited on signal 11

What version of the product are you using? On what operating system?
Latest trunk, SLES 10

Please provide any additional information below.
Can you please have a look at the nginx forum? Thank you very much.
http://forum.nginx.org/read.php?2,225420

Original issue reported on code.google.com by Marc.Sch...@gmail.com on 19 Apr 2012 at 5:32

Answer 1 · 2015-04-09T02:10:36.000Z

Hi, I'll check this problem. 

Can you show me your configure file and the page caused this problem? This is a 
obscure bug. I want to reproduce it in my box.

There is an known issue. I find my module may encounter problem when processing 
some very huge lines in proxy buffering mode.

The methods below may help you to avoid this bug:
1. Can you turn the buffering off?
proxy_buffering off;

2. Or you can enlarge the proxy busy buffer size:
proxy_buffers           128 16k;

Hope this can alleviate your problem.

Original comment by nbubi...@gmail.com on 19 Apr 2012 at 9:28

Answer 2 · 2015-04-09T02:10:36.000Z

Hi, thanks.

The problem seems to occur only when using the fastcgi backend. The traced core 
file lead to a pdf generation via php.

subs_filter_types text/html text/css application/x-javascript;
then some regexp like
subs_filter (((https?|file):\/\/)(\S*))?(\/*)fileadmin\/DAM\/ 
$SSL_PROTO://domain.com/fileadmin/DAM/ ri;
subs_filter (((https?|file):\/\/)(\S*))?(\/*)txt3233.gif 
$CDN_DOMAIN2/txt3233.gif gi;

My config:
fastcgi_intercept_errors        on;
fastcgi_ignore_client_abort     off;
fastcgi_connect_timeout         90;
fastcgi_send_timeout            180;
fastcgi_read_timeout            180;
fastcgi_buffer_size             256k;
fastcgi_buffers                 10 256k;
fastcgi_busy_buffers_size       256k;

Buffers are already high, but maybe not high enough for the pdf content...

Original comment by Marc.Sch...@gmail.com on 19 Apr 2012 at 9:40

Answer 3 · 2015-04-09T02:10:36.000Z

[deleted comment]

Answer 4 · 2015-04-09T02:10:36.000Z

I can see it parsing the pdf content in debuglog.


2012/04/19 11:46:46 [debug] 7563#0: *196 http fastcgi parser: 0
2012/04/19 11:46:46 [debug] 7563#0: *196 http fastcgi header: "Content-Length: 
355230"
2012/04/19 11:46:46 [debug] 7563#0: *196 http fastcgi parser: 0
2012/04/19 11:46:46 [debug] 7563#0: *196 http fastcgi header: 
"Content-disposition: attachment; filename=filename.pdf"
2012/04/19 11:46:46 [debug] 7563#0: *196 http fastcgi parser: 0
2012/04/19 11:46:46 [debug] 7563#0: *196 http fastcgi header: "Content-Type: 
text/html; charset=utf-8"
2012/04/19 11:46:46 [debug] 7563#0: *196 http fastcgi parser: 0
2012/04/19 11:46:46 [debug] 7563#0: *196 http fastcgi header: "Cache-Control: 
private"
2012/04/19 11:46:46 [debug] 7563#0: *196 http fastcgi parser: 1
2012/04/19 11:46:46 [debug] 7563#0: *196 http fastcgi header done
2012/04/19 11:46:46 [debug] 7563#0: *196 http subs filter header "/index.php"
2012/04/19 11:46:46 [debug] 7563#0: *196 posix_memalign: 000000000069DEF0:4096 
@16
2012/04/19 11:46:46 [debug] 7563#0: *196 headers more header filter, uri 
"/index.php"



2012/04/19 11:46:46 [debug] 7563#0: *196 http subs filter start: 
"match:(((https?|file):\/\/)(\S*))?(\/*)emp\/GB\/, sub:http://domain/emp/GB/ 
regex:1, dup_capture:0"
2012/04/19 11:46:46 [debug] 7563#0: *196 current line in: 
"xœ½WÛnÛF�}÷WÌc�Dë½“Û<ù�¨�ÝÖQ‹�M�(r%±¡I™¤bäCú�ùÆÎ’
Ë�J)GV„J€¨åeçœ™3�røñ„��ÀãÉùN¯�0F(…Ù�®f'î/…åÉ�(¼I
�Å¯ÒD��( ��p�*ˆïátÊà²€_q3ÃñF"Ýí„2
wop�&:��˜�X�šP
A��@Á=Hfˆ�Ý:ƒwð€6—€�2��C!0œP�³�^üR�É&®aš/Šò>ªÓ"� ³¿q_��
��ã[t¶7¿�¥8afÔ˜äœ�Ý�3ÞX´´À X

Original comment by Marc.Sch...@gmail.com on 19 Apr 2012 at 10:03

Answer 5 · 2015-04-09T02:10:36.000Z

Yes, can you attach the full debug.log when coredump? Or you can give me a pdf 
file which can cause core dump.

Thanks.

Original comment by nbubi...@gmail.com on 19 Apr 2012 at 10:05

Answer 6 · 2015-04-09T02:10:36.000Z

Hi, attached you can find the file. When i try to reproduce it, the error 
doesn't appear. It seems to be client-dependant, don't know why.

Original comment by Marc.Sch...@gmail.com on 19 Apr 2012 at 10:17

Attachments:

error_log.zip

Answer 7 · 2015-04-09T02:10:37.000Z

Thanks. I'll review this log.

Original comment by nbubi...@gmail.com on 19 Apr 2012 at 10:30

Answer 8 · 2015-04-09T02:10:37.000Z

Can you try the code in this branch:
https://github.com/yaoweibin/ngx_http_substitutions_filter_module/tree/developme
nt

Hope this can help you.

I'm just moving this module to the github. It's difficult to create a branch 
with subversion.

Thanks.

Original comment by nbubi...@gmail.com on 23 Apr 2012 at 3:40

Answer 9 · 2015-04-09T02:10:37.000Z

Thank you. It's in test, i'll report asap.

Original comment by Marc.Sch...@gmail.com on 24 Apr 2012 at 5:05

Answer 10 · 2015-04-09T02:10:37.000Z

Still no luck, but i'm not sure if it is caused by this module.

Core was generated by `nginx: worker pr'.
Program terminated with signal 11, Segmentation fault.
#0  ngx_event_pipe_add_free_buf (p=0x767698, b=0x0) at 
src/event/ngx_event_pipe.c:949
949     src/event/ngx_event_pipe.c: No such file or directory.
        in src/event/ngx_event_pipe.c
(gdb) backtrace
#0  ngx_event_pipe_add_free_buf (p=0x767698, b=0x0) at 
src/event/ngx_event_pipe.c:949
#1  0x0000000000429f73 in ngx_event_pipe (p=0x767698, do_write=<value optimized 
out>) at src/event/ngx_event_pipe.c:667
#2  0x000000000045168e in ngx_http_upstream_process_upstream (r=0x72ab50, 
u=0x7672e0) at src/http/ngx_http_upstream.c:2641
#3  0x00000000004517e9 in ngx_http_upstream_handler (ev=0x2b9a0592a420) at 
src/http/ngx_http_upstream.c:935
#4  0x00000000004314e3 in ngx_epoll_process_events (cycle=0x5de490, 
timer=<value optimized out>, flags=<value optimized out>) at 
src/event/modules/ngx_epoll_module.c:679
#5  0x000000000042797e in ngx_process_events_and_timers (cycle=0x5de490) at 
src/event/ngx_event.c:247
#6  0x000000000042e966 in ngx_worker_process_cycle (cycle=0x5de490, data=<value 
optimized out>) at src/os/unix/ngx_process_cycle.c:806
#7  0x000000000042ce7d in ngx_spawn_process (cycle=0x5de490, proc=0x42e8a0 
<ngx_worker_process_cycle>, data=0x0, name=0x48c736 "worker process", 
respawn=-3) at src/os/unix/ngx_process.c:198
#8  0x000000000042deac in ngx_start_worker_processes (cycle=0x5de490, n=8, 
type=-3) at src/os/unix/ngx_process_cycle.c:365
#9  0x000000000042f0bf in ngx_master_process_cycle (cycle=0x5de490) at 
src/os/unix/ngx_process_cycle.c:137
#10 0x0000000000410fdd in main (argc=1, argv=<value optimized out>) at 
src/core/nginx.c:410
(gdb) backtrace full
#0  ngx_event_pipe_add_free_buf (p=0x767698, b=0x0) at 
src/event/ngx_event_pipe.c:949
        cl = (ngx_chain_t *) 0x73f340
#1  0x0000000000429f73 in ngx_event_pipe (p=0x767698, do_write=<value optimized 
out>) at src/event/ngx_event_pipe.c:667
        rc = <value optimized out>
        rev = <value optimized out>
        wev = <value optimized out>
#2  0x000000000045168e in ngx_http_upstream_process_upstream (r=0x72ab50, 
u=0x7672e0) at src/http/ngx_http_upstream.c:2641
        c = (ngx_connection_t *) 0x2b99c3325790
#3  0x00000000004517e9 in ngx_http_upstream_handler (ev=0x2b9a0592a420) at 
src/http/ngx_http_upstream.c:935
        c = (ngx_connection_t *) 0x2b99c3325fd0
        r = (ngx_http_request_t *) 0x72ab50
        u = (ngx_http_upstream_t *) 0x7672e0
#4  0x00000000004314e3 in ngx_epoll_process_events (cycle=0x5de490, 
timer=<value optimized out>, flags=<value optimized out>) at 
src/event/modules/ngx_epoll_module.c:679
        events = <value optimized out>
        revents = 5
        instance = 1
        i = 0
        level = <value optimized out>
        err = <value optimized out>
        rev = (ngx_event_t *) 0x2b9a0592a420
        wev = <value optimized out>
        queue = <value optimized out>
        c = (ngx_connection_t *) 0x2b99c3325790
#5  0x000000000042797e in ngx_process_events_and_timers (cycle=0x5de490) at 
src/event/ngx_event.c:247
        flags = 1
        timer = 500
        delta = 1335254598933
#6  0x000000000042e966 in ngx_worker_process_cycle (cycle=0x5de490, data=<value 
optimized out>) at src/os/unix/ngx_process_cycle.c:806
        i = 0
        c = <value optimized out>
#7  0x000000000042ce7d in ngx_spawn_process (cycle=0x5de490, proc=0x42e8a0 
<ngx_worker_process_cycle>, data=0x0, name=0x48c736 "worker process", 
respawn=-3) at src/os/unix/ngx_process.c:198
        on = 1
        pid = 0
        s = 1
#8  0x000000000042deac in ngx_start_worker_processes (cycle=0x5de490, n=8, 
type=-3) at src/os/unix/ngx_process_cycle.c:365
        i = 2
        ch = {command = 1, pid = 3149, slot = 0, fd = 3}
#9  0x000000000042f0bf in ngx_master_process_cycle (cycle=0x5de490) at 
src/os/unix/ngx_process_cycle.c:137
        title = 0x730b08 "master process /usr/sbin/nginx"
        p = <value optimized out>
        size = 31
        i = 1
        n = <value optimized out>
        sigio = <value optimized out>
        set = {__val = {0 <repeats 16 times>}}
        itv = {it_interval = {tv_sec = 0, tv_usec = 6153384}, it_value = {tv_sec = 0, tv_usec = 47939406459440}}
        live = <value optimized out>
        delay = <value optimized out>
        ccf = (ngx_core_conf_t *) 0x5df470
#10 0x0000000000410fdd in main (argc=1, argv=<value optimized out>) at 
src/core/nginx.c:410
        i = <value optimized out>
        log = (ngx_log_t *) 0x5b6ea0
        cycle = (ngx_cycle_t *) 0x5de490
        init_cycle = {conf_ctx = 0x0, pool = 0x5dd440, log = 0x5b6ea0, new_log = {log_level = 0, file = 0x0, connection = 0, handler = 0, data = 0x0, action = 0x0}, files = 0x0,
  free_connections = 0x0, free_connection_n = 0, reusable_connections_queue = {prev = 0x0, next = 0x0}, listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, pathes = {
    elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, open_files = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, shared_memory = {
    last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0, files_n = 0, connections = 0x0, read_events = 0x0, write_events = 0x0,
  old_cycle = 0x0, conf_file = {len = 21, data = 0x488072 "/etc/nginx/nginx.conf"}, conf_param = {len = 0, data = 0x0}, conf_prefix = {len = 11, data = 0x488072 "/etc/nginx/nginx.conf"},
  prefix = {len = 6, data = 0x48806b "/usr//"}, lock_file = {len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
        ccf = (ngx_core_conf_t *) 0x5df470

Original comment by Marc.Sch...@gmail.com on 24 Apr 2012 at 8:21

Answer 11 · 2015-04-09T02:10:37.000Z

I think it should be caused by my module. But it's not very explicit. I'll try 
other ways. 

Thanks.

Original comment by nbubi...@gmail.com on 24 Apr 2012 at 1:44

Answer 12 · 2015-04-09T02:10:38.000Z

Can you try the latest revision? 
https://github.com/yaoweibin/ngx_http_substitutions_filter_module

I have refactored this module. Now the module is more readable and fit with the 
latest Nginx code.

Hope this can help you. This is a development version, you should test it 
carefully.

Thanks

Original comment by yaoweibin@gmail.com on 30 Jun 2012 at 2:53