ngx-leaked-pools breaks against openresty 1.4.2.9
rohityadavcloud opened this issue · 5 comments
Hi, just want to report that some of the tools of this repo are not working for me;
root@systemtap:~/nginx-systemtap-toolkit# ./ngx-leaked-pools -p 146241
semantic error: while resolving probe point: identifier 'process' at :9:7
source: probe process("/var/www/dev/bin/nginx").provider("nginx").mark("create-pool-done")
My setup has openresty 1.4.2.9 (debug) on Ubuntu 12.04.3/Linux 3.8 with systemtap 2.3
other fails; ngx-shm failed to work against one of the works on a zone, I see shared lua dictionary is not available on all worker processes, will mail asking about it on the openresty ML. Thanks.
The ngx-leaked-pools requires the dtrace patch for nginx. To quote the documentation for ngx-leaked-pools:
"This script requires Nginx instances that have applied the latest dtrace patch. See the dtrace-nginx project for more details.
The upcoming 1.2.3.3 release of the ngx_openresty will include the right dtrace patch by default. And you just need to build it with the --with-dtrace-probes configure option."
For the ngx-shm tool failures, please provide the original error messages. One common thing is page faults due to the fact that the worker you traced had never actually read that zone yet.
BTW, don't build ngx_openresty with --with-debug for production because it has significant performance penalty. None of my tracing tools depend on --with-debug.
You can safely enable --with-dtrace-probes for production though.
Thank you for the suggestion, will try with that. I found it by experimenting that --with-debug has performance penalties, makes the web server slower that also ends up consuming some more memory.
About zones, page faults and tracing workers, I was exploring how nginx's memory management [1] which ngx shared dict uses too as you suggested on ML. A SIGHUP based reload confirmed that the shared dict was actually shared using virtual memory so while I was wondering why workers don't just share pages [2] like other linux memory [3] models (have gone through some theory only), is it that nginx/openresty workers tend to deviate from that and have their own copies in physical memory where they could have shared physical memory? Does nginx's own allocation model wrt workers, relate to mmap, page tables, shared memory via virtualmemory/paging?
[1] http://www.evanmiller.org/nginx-modules-guide-advanced.html
[2] http://www.tldp.org/LDP/tlk/mm/memory.html
[3] http://www.cs.uic.edu/~jbell/CourseNotes/OperatingSystems/9_VirtualMemory.html
Hello!
On Thu, Oct 10, 2013 at 2:56 PM, Bhaisaab wrote:
About zones, page faults and tracing workers, I was exploring how nginx's memory management [1] which ngx shared dict uses too as you suggested on ML.
ngx_lua is just accessing the shared memory zones directly. The shared
memory zones are created by the master process via the mmap() syscall.
And all the worker processes automatically inherit the shm zones via
fork() in master.
A SIGHUP based reload confirmed that the shared dict was actually shared using virtual memory so while I was wondering why workers don't just share pages [2] like other linux memory [3] models (have gone through some theory only),
The memory pages are only loaded into the forked child processes on
demand. It is called "demand paging". After HUP reload, the old child
processes exit and new child processes are spawned. Because the newly
created child processes have never access any memory pages in the shm
zones, the memory metric you read from ps or top is small.
is it that nginx/openresty workers tend to deviate from that and
have their own copies in physical memory where they could have
shared physical memory?
No. My email in the mailing list already makes it clear that page
loading happens here, but not page copying.
Does nginx's own allocation model wrt workers, relate to mmap,
page tables, shared memory via virtualmemory/paging?
Only mmap() and fork() are used here for shared memory things.
To better understand the memory usage of your nginx worker processes.
Use the pmap command instead. For example:
pmap -d <worker-pid>
And look at the last line of the output. For example, on my side, with
an nginx worker with VIRT 151m, RES 102m, and SHR 101m, we have the
following line in pmap's output:
mapped: 155624K writeable/private: 1716K shared: 102404K
The writeable/private value is the size of memory the worker process
alone takes. We can see that shared memory pages are not copied to
the process space, they're just loaded into it, addding huge values
to RES but almost nothing to the physical memory.
Regards,
-agentzh
Thanks agentzh for the information, it helps to learn these details. Will close the issue.