query domains in a for loop
Closed this issue · 8 comments
mayuroks commented
I have tried it but everytime I rub into a different issue.
Can you gimme a simple example where If I had and array of domains how would i query them using a for loop ?
ioquatix commented
mayuroks commented
Tried the above solution but landed into this error
/usr/local/rvm/gems/ruby-1.9.3-p545/gems/eventmachine-1.0.3/lib/eventmachine.rb:664:in `connect_server': unable to create new socket: Too many open files (EventMachine::ConnectionError)
from /usr/local/rvm/gems/ruby-1.9.3-p545/gems/eventmachine-1.0.3/lib/eventmachine.rb:664:in `bind_connect'
from /usr/local/rvm/gems/ruby-1.9.3-p545/gems/eventmachine-1.0.3/lib/eventmachine.rb:640:in `connect'
from /usr/local/rvm/gems/ruby-1.9.3-p545/gems/rubydns-0.5.0/lib/rubydns/resolver.rb:175:in `open'
from /usr/local/rvm/gems/ruby-1.9.3-p545/gems/rubydns-0.5.0/lib/rubydns/resolver.rb:127:in `try_next_server!'
from /usr/local/rvm/gems/ruby-1.9.3-p545/gems/rubydns-0.5.0/lib/rubydns/resolver.rb:136:in `block in try_next_server!'
from /usr/local/rvm/gems/ruby-1.9.3-p545/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `call'
from /usr/local/rvm/gems/ruby-1.9.3-p545/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run_machine'
from /usr/local/rvm/gems/ruby-1.9.3-p545/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run'
from ./rubydns_test.rb:13:in `<main>'Please note I have around thousands of domains to query.
ioquatix commented
The problem is you have too many open files. If you are on Mac, you need to increase the limit.. ulimit -n 20000
mayuroks commented
hmm I am using ubuntu, just curious If the program open so many file descriptors does it clean those later ?
mayuroks commented
Increased ulimit -n from 1024 to 50000 now got buffer overflow error
*** buffer overflow detected ***: ruby terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x741cf)[0x7f5f7cfe91cf]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f5f7d080b2c]
/lib/x86_64-linux-gnu/libc.so.6(+0x10a9f0)[0x7f5f7d07f9f0]
/lib/x86_64-linux-gnu/libc.so.6(+0x10ba77)[0x7f5f7d080a77]
/usr/local/rvm/gems/ruby-1.9.3-p545/extensions/x86_64-linux/1.9.1/eventmachine-1.0.3/rubyeventmachine.so(_ZN14EventMachine_t14_RunSelectOnceEv+0xe5)[0x7f5f7aa69e25]
/usr/local/rvm/gems/ruby-1.9.3-p545/extensions/x86_64-linux/1.9.1/eventmachine-1.0.3/rubyeventmachine.so(_ZN14EventMachine_t8_RunOnceEv+0x1b)[0x7f5f7aa6aceb]
/usr/local/rvm/gems/ruby-1.9.3-p545/extensions/x86_64-linux/1.9.1/eventmachine-1.0.3/rubyeventmachine.so(_ZN14EventMachine_t3RunEv+0x58)[0x7f5f7aa6cd28]
/usr/local/rvm/gems/ruby-1.9.3-p545/extensions/x86_64-linux/1.9.1/eventmachine-1.0.3/rubyeventmachine.so(+0xf9a9)[0x7f5f7aa639a9]
/usr/local/rvm/rubies/ruby-1.9.3-p545/lib/libruby.so.1.9(+0x1948a3)[0x7f5f7d4cf8a3]
/usr/local/rvm/rubies/ruby-1.9.3-p545/lib/libruby.so.1.9(+0x18ddad)[0x7f5f7d4c8dad]
/usr/local/rvm/rubies/ruby-1.9.3-p545/lib/libruby.so.1.9(+0x192038)[0x7f5f7d4cd038]
/usr/local/rvm/rubies/ruby-1.9.3-p545/lib/libruby.so.1.9(rb_iseq_eval_main+0x9f)[0x7f5f7d4d5e7f]
/usr/local/rvm/rubies/ruby-1.9.3-p545/lib/libruby.so.1.9(+0x6a13a)[0x7f5f7d3a513a]
/usr/local/rvm/rubies/ruby-1.9.3-p545/lib/libruby.so.1.9(ruby_exec_node+0x1d)[0x7f5f7d3a606d]
/usr/local/rvm/rubies/ruby-1.9.3-p545/lib/libruby.so.1.9(ruby_run_node+0x1c)[0x7f5f7d3a82cc]
ruby[0x40089b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f5f7cf96ec5]
ruby[0x4008c9]
======= Memory map: ========
00400000-00401000 r-xp 00000000 08:01 1589665 /usr/local/rvm/rubies/ruby-1.9.3-p545/bin/ruby
00600000-00601000 r--p 00000000 08:01 1589665 /usr/local/rvm/rubies/ruby-1.9.3-p545/bin/ruby
00601000-00602000 rw-p 00001000 08:01 1589665 /usr/local/rvm/rubies/ruby-1.9.3-p545/bin/ruby
013b0000-020fe000 rw-p 00000000 00:00 0 [heap]
7f5f7a12e000-7f5f7a131000 r-xp 00000000 08:01 1590008 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/etc.so
7f5f7a131000-7f5f7a330000 ---p 00003000 08:01 1590008 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/etc.so
7f5f7a330000-7f5f7a331000 r--p 00002000 08:01 1590008 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/etc.so
7f5f7a331000-7f5f7a332000 rw-p 00003000 08:01 1590008 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/etc.so
7f5f7a332000-7f5f7a339000 r-xp 00000000 08:01 1590006 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/pathname.so
7f5f7a339000-7f5f7a538000 ---p 00007000 08:01 1590006 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/pathname.so
7f5f7a538000-7f5f7a539000 r--p 00006000 08:01 1590006 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/pathname.so
7f5f7a539000-7f5f7a53a000 rw-p 00007000 08:01 1590006 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/pathname.so
7f5f7a53a000-7f5f7a550000 r-xp 00000000 08:01 1704700 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5f7a550000-7f5f7a74f000 ---p 00016000 08:01 1704700 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5f7a74f000-7f5f7a750000 rw-p 00015000 08:01 1704700 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5f7a750000-7f5f7a836000 r-xp 00000000 08:01 134702 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
7f5f7a836000-7f5f7aa35000 ---p 000e6000 08:01 134702 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
7f5f7aa35000-7f5f7aa3d000 r--p 000e5000 08:01 134702 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
7f5f7aa3d000-7f5f7aa3f000 rw-p 000ed000 08:01 134702 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
7f5f7aa3f000-7f5f7aa54000 rw-p 00000000 00:00 0
7f5f7aa54000-7f5f7aa7b000 r-xp 00000000 08:01 920815 /usr/local/rvm/gems/ruby-1.9.3-p545/extensions/x86_64-linux/1.9.1/eventmachine-1.0.3/rubyeventmachine.so
7f5f7aa7b000-7f5f7ac7b000 ---p 00027000 08:01 920815 /usr/local/rvm/gems/ruby-1.9.3-p545/extensions/x86_64-linux/1.9.1/eventmachine-1.0.3/rubyeventmachine.so
7f5f7ac7b000-7f5f7ac7c000 r--p 00027000 08:01 920815 /usr/local/rvm/gems/ruby-1.9.3-p545/extensions/x86_64-linux/1.9.1/eventmachine-1.0.3/rubyeventmachine.so
7f5f7ac7c000-7f5f7ac7e000 rw-p 00028000 08:01 920815 /usr/local/rvm/gems/ruby-1.9.3-p545/extensions/x86_64-linux/1.9.1/eventmachine-1.0.3/rubyeventmachine.so
7f5f7ac7e000-7f5f7ac85000 r-xp 00000000 08:01 1589992 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/stringio.so
7f5f7ac85000-7f5f7ae84000 ---p 00007000 08:01 1589992 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/stringio.so
7f5f7ae84000-7f5f7ae85000 r--p 00006000 08:01 1589992 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/stringio.so
7f5f7ae85000-7f5f7ae86000 rw-p 00007000 08:01 1589992 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/stringio.so
7f5f7ae86000-7f5f7ae89000 r-xp 00000000 08:01 1589993 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/digest.so
7f5f7ae89000-7f5f7b089000 ---p 00003000 08:01 1589993 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/digest.so
7f5f7b089000-7f5f7b08a000 r--p 00003000 08:01 1589993 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/digest.so
7f5f7b08a000-7f5f7b08b000 rw-p 00004000 08:01 1589993 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/digest.so
7f5f7b08b000-7f5f7b23c000 r-xp 00000000 08:01 1704720 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f5f7b23c000-7f5f7b43b000 ---p 001b1000 08:01 1704720 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f5f7b43b000-7f5f7b456000 r--p 001b0000 08:01 1704720 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f5f7b456000-7f5f7b461000 rw-p 001cb000 08:01 1704720 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f5f7b461000-7f5f7b465000 rw-p 00000000 00:00 0
7f5f7b465000-7f5f7b4b9000 r-xp 00000000 08:01 1704719 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f5f7b4b9000-7f5f7b6b9000 ---p 00054000 08:01 1704719 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f5f7b6b9000-7f5f7b6bc000 r--p 00054000 08:01 1704719 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f5f7b6bc000-7f5f7b6c2000 rw-p 00057000 08:01 1704719 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f5f7b6c2000-7f5f7b6c3000 rw-p 00000000 00:00 0
7f5f7b6c3000-7f5f7b711000 r-xp 00000000 08:01 1590004 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/openssl.so
7f5f7b711000-7f5f7b910000 ---p 0004e000 08:01 1590004 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/openssl.so
7f5f7b910000-7f5f7b911000 r--p 0004d000 08:01 1590004 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/openssl.so
7f5f7b911000-7f5f7b913000 rw-p 0004e000 08:01 1590004 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/openssl.so
7f5f7b913000-7f5f7b914000 rw-p 00000000 00:00 0
7f5f7b914000-7f5f7b915000 r-xp 00000000 08:01 1589989 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/fcntl.so
7f5f7b915000-7f5f7bb14000 ---p 00001000 08:01 1589989 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/fcntl.so
7f5f7bb14000-7f5f7bb15000 r--p 00000000 08:01 1589989 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/fcntl.so
7f5f7bb15000-7f5f7bb16000 rw-p 00001000 08:01 1589989 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/fcntl.so
7f5f7bb16000-7f5f7bb3b000 r-xp 00000000 08:01 1589984 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/socket.so
7f5f7bb3b000-7f5f7bd3a000 ---p 00025000 08:01 1589984 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/socket.so
7f5f7bd3a000-7f5f7bd3b000 r--p 00024000 08:01 1589984 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/socket.so
7f5f7bd3b000-7f5f7bd3c000 rw-p 00025000 08:01 1589984 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/socket.so
7f5f7bd3c000-7f5f7bd3e000 r-xp 00000000 08:01 2114274 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7f5f7bd3e000-7f5f7bf3e000 ---p 00002000 08:01 2114274 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7f5f7bf3e000-7f5f7bf3f000 r--p 00002000 08:01 2114274 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7f5f7bf3f000-7f5f7bf40000 rw-p 00003000 08:01 2114274 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7f5f7bf40000-7f5f7bf42000 r-xp 00000000 08:01 2114253 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7f5f7bf42000-7f5f7c141000 ---p 00002000 08:01 2114253 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7f5f7c141000-7f5f7c142000 r--p 00001000 08:01 2114253 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7f5f7c142000-7f5f7c143000 rw-p 00002000 08:01 2114253 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7f5f7c143000-7f5f7c40c000 r--p 00000000 08:01 131045 /usr/lib/locale/locale-archive
7f5f7c40c000-7f5f7c511000 r-xp 00000000 08:01 1705139 /lib/x86_64-linux-gnu/libm-2.19.so
7f5f7c511000-7f5f7c710000 ---p 00105000 08:01 1705139 /lib/x86_64-linux-gnu/libm-2.19.so
7f5f7c710000-7f5f7c711000 r--p 00104000 08:01 1705139 /lib/x86_64-linux-gnu/libm-2.19.so
7f5f7c711000-7f5f7c712000 rw-p 00105000 08:01 1705139 /lib/x86_64-linux-gnu/libm-2.19.so
7f5f7c712000-7f5f7c71b000 r-xp 00000000 08:01 1705155 /lib/x86_64-linux-gnu/libcrypt-2.19.so
7f5f7c71b000-7f5f7c91b000 ---p 00009000 08:01 1705155 /lib/x86_64-linux-gnu/libcrypt-2.19.so
7f5f7c91b000-7f5f7c91c000 r--p 00009000 08:01 1705155 /lib/x86_64-linux-gnu/libcrypt-2.19.so
7f5f7c91c000-7f5f7c91d000 rw-p 0000a000 08:01 1705155 /lib/x86_64-linux-gnu/libcrypt-2.19.so
7f5f7c91d000-7f5f7c94b000 rw-p 00000000 00:00 0
7f5f7c94b000-7f5f7c94e000 r-xp 00000000 08:01 1705159 /lib/x86_64-linux-gnu/libdl-2.19.so
7f5f7c94e000-7f5f7cb4d000 ---p 00003000 08:01 1705159 /lib/x86_64-linux-gnu/libdl-2.19.so
7f5f7cb4d000-7f5f7cb4e000 r--p 00002000 08:01 1705159 /lib/x86_64-linux-gnu/libdl-2.19.so
7f5f7cb4e000-7f5f7cb4f000 rw-p 00003000 08:01 1705159 /lib/x86_64-linux-gnu/libdl-2.19.so
7f5f7cb4f000-7f5f7cb56000 r-xp 00000000 08:01 1705165 /lib/x86_64-linux-gnu/librt-2.19.so
7f5f7cb56000-7f5f7cd55000 ---p 00007000 08:01 1705165 /lib/x86_64-linux-gnu/librt-2.19.so
7f5f7cd55000-7f5f7cd56000 r--p 00006000 08:01 1705165 /lib/x86_64-linux-gnu/librt-2.19.so
7f5f7cd56000-7f5f7cd57000 rw-p 00007000 08:01 1705165 /lib/x86_64-linux-gnu/librt-2.19.so
7f5f7cd57000-7f5f7cd70000 r-xp 00000000 08:01 1705164 /lib/x86_64-linux-gnu/libpthread-2.19.so
7f5f7cd70000-7f5f7cf6f000 ---p 00019000 08:01 1705164 /lib/x86_64-linux-gnu/libpthread-2.19.so
7f5f7cf6f000-7f5f7cf70000 r--p 00018000 08:01 1705164 /lib/x86_64-linux-gnu/libpthread-2.19.so
7f5f7cf70000-7f5f7cf71000 rw-p 00019000 08:01 1705164 /lib/x86_64-linux-gnu/libpthread-2.19.so
7f5f7cf71000-7f5f7cf75000 rw-p 00000000 00:00 0
7f5f7cf75000-7f5f7d131000 r-xp 00000000 08:01 1705133 /lib/x86_64-linux-gnu/libc-2.19.so
7f5f7d131000-7f5f7d330000 ---p 001bc000 08:01 1705133 /lib/x86_64-linux-gnu/libc-2.19.so
7f5f7d330000-7f5f7d334000 r--p 001bb000 08:01 1705133 /lib/x86_64-linux-gnu/libc-2.19.so
7f5f7d334000-7f5f7d336000 rw-p 001bf000 08:01 1705133 /lib/x86_64-linux-gnu/libc-2.19.so
7f5f7d336000-7f5f7d33b000 rw-p 00000000 00:00 0
7f5f7d33b000-7f5f7d580000 r-xp 00000000 08:01 1589667 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/libruby.so.1.9.1
7f5f7d580000-7f5f7d77f000 ---p 00245000 08:01 1589667 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/libruby.so.1.9.1
7f5f7d77f000-7f5f7d784000 r--p 00244000 08:01 1589667 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/libruby.so.1.9.1
7f5f7d784000-7f5f7d788000 rw-p 00249000 08:01 1589667 /usr/local/rvm/rubies/ruby-1.9.3-p545/lib/libruby.so.1.9.1
7f5f7d788000-7f5f7d7a5000 rw-p 00000000 00:00 0
7f5f7d7a5000-7f5f7d7c8000 r-xp 00000000 08:01 1705131 /lib/x86_64-linux-gnu/ld-2.19.so
7f5f7d89f000-7f5f7d9a5000 rw-p 00000000 00:00 0
7f5f7d9c0000-7f5f7d9c1000 rw-p 00000000 00:00 0
7f5f7d9c1000-7f5f7d9c2000 ---p 00000000 00:00 0
7f5f7d9c2000-7f5f7d9c7000 rw-p 00000000 00:00 0 [stack:30535]
7f5f7d9c7000-7f5f7d9c8000 r--p 00022000 08:01 1705131 /lib/x86_64-linux-gnu/ld-2.19.so
7f5f7d9c8000-7f5f7d9c9000 rw-p 00023000 08:01 1705131 /lib/x86_64-linux-gnu/ld-2.19.so
7f5f7d9c9000-7f5f7d9ca000 rw-p 00000000 00:00 0
7fff6e5b6000-7fff6e5d7000 rw-p 00000000 00:00 0 [stack]
7fff6e5fe000-7fff6e600000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Aborted (core dumped)ioquatix commented
I actually found this issue too. Firstly, the best option is to batch the requests, because if you make too many requests there aren't enough sockets and the remote server probably can't handle them all. Secondly, I'm currently experimenting with a fork which uses Celluloid instead of EventMachine since it seems like EventMachine is now unmaintained. The immediate solution is to batch the requests in groups of 100, and hopefully in the future the resolver will be more robust.
mayuroks commented
OK I will give it a try. Thanks for helping