Logstash stuck while loading GeoIP DB

[adsanz]

So, I followed the proccess as drescribed, and when I try to run logstash and check the log it get stuck on this point, been more than 20 mins so I just thought it may coul be some kind of issue:

The issue I talk about is at the end of the log file, but I add everything if needed.

[2018-12-28T23:14:33,717][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} [2018-12-28T23:14:33,724][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} [2018-12-28T23:14:34,203][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.4"} [2018-12-28T23:14:34,324][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2018-12-28T23:14:59,212][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"synlite_suricata", "pipeline.workers"=>3, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} [2018-12-28T23:14:59,577][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://192.168.31.125:9200/]}} [2018-12-28T23:14:59,580][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://192.168.31.125:9200/, :path=>"/"} [2018-12-28T23:14:59,720][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://192.168.31.125:9200/"} [2018-12-28T23:14:59,772][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6} [2018-12-28T23:14:59,772][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the typeevent field won't be used to determine the document _type {:es_version=>6} [2018-12-28T23:14:59,788][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/etc/logstash/synlite_suricata/templates/synlite_suricata_stats.template.json"} [2018-12-28T23:14:59,799][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"order"=>0, "version"=>10001, "index_patterns"=>"suricata_stats-1.0.1-*", "settings"=>{"index"=>{"number_of_shards"=>3, "number_of_replicas"=>1, "refresh_interval"=>"10s", "codec"=>"best_compression"}}, "mappings"=>{"_default_"=>{"numeric_detection"=>true, "dynamic_templates"=>[{"string_fields"=>{"match_mapping_type"=>"string", "match"=>"*", "mapping"=>{"type"=>"keyword"}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "event"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"host"=>{"type"=>"keyword"}, "subtype"=>{"type"=>"keyword"}, "type"=>{"type"=>"keyword"}}}, "node"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ipaddr"=>{"type"=>"ip"}, "hostname"=>{"type"=>"keyword"}}}, "stats"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"app_layer"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"flow"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"dcerpc_tcp"=>{"type"=>"long"}, "dcerpc_udp"=>{"type"=>"long"}, "dnp3"=>{"type"=>"long"}, "dns_tcp"=>{"type"=>"long"}, "dns_udp"=>{"type"=>"long"}, "failed_tcp"=>{"type"=>"long"}, "failed_udp"=>{"type"=>"long"}, "ftp"=>{"type"=>"long"}, "http"=>{"type"=>"long"}, "imap"=>{"type"=>"long"}, "msn"=>{"type"=>"long"}, "modbus"=>{"type"=>"long"}, "smb"=>{"type"=>"long"}, "smtp"=>{"type"=>"long"}, "ssh"=>{"type"=>"long"}, "tls"=>{"type"=>"long"}}}, "tx"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"dcerpc_tcp"=>{"type"=>"long"}, "dcerpc_udp"=>{"type"=>"long"}, "dnp3"=>{"type"=>"long"}, "dns_tcp"=>{"type"=>"long"}, "dns_udp"=>{"type"=>"long"}, "ftp"=>{"type"=>"long"}, "http"=>{"type"=>"long"}, "modbus"=>{"type"=>"long"}, "smb"=>{"type"=>"long"}, "smtp"=>{"type"=>"long"}, "ssh"=>{"type"=>"long"}, "tls"=>{"type"=>"long"}}}}}, "capture"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"kernel_drops"=>{"type"=>"long"}, "kernel_packets"=>{"type"=>"long"}}}, "defrag"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ipv4"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"fragments"=>{"type"=>"long"}, "reassembled"=>{"type"=>"long"}, "timeouts"=>{"type"=>"long"}}}, "ipv6"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"fragments"=>{"type"=>"long"}, "reassembled"=>{"type"=>"long"}, "timeouts"=>{"type"=>"long"}}}, "max_frag_hits"=>{"type"=>"long"}}}, "decoder"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"avg_pkt_size"=>{"type"=>"long"}, "bytes"=>{"type"=>"long"}, "dce"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"pkt_too_small"=>{"type"=>"long"}}}, "erspan"=>{"type"=>"long"}, "ethernet"=>{"type"=>"long"}, "gre"=>{"type"=>"long"}, "icmpv4"=>{"type"=>"long"}, "icmpv6"=>{"type"=>"long"}, "ieee8021ah"=>{"type"=>"long"}, "invalid"=>{"type"=>"long"}, "ipraw"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"invalid_ip_version"=>{"type"=>"long"}}}, "ipv4"=>{"type"=>"long"}, "ipv4_in_ipv6"=>{"type"=>"long"}, "ipv6"=>{"type"=>"long"}, "ipv6_in_ipv6"=>{"type"=>"long"}, "ltnull"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"pkt_too_small"=>{"type"=>"long"}, "unsupported_type"=>{"type"=>"long"}}}, "max_pkt_size"=>{"type"=>"long"}, "mpls"=>{"type"=>"long"}, "null"=>{"type"=>"long"}, "pkts"=>{"type"=>"long"}, "ppp"=>{"type"=>"long"}, "pppoe"=>{"type"=>"long"}, "raw"=>{"type"=>"long"}, "sctp"=>{"type"=>"long"}, "sll"=>{"type"=>"long"}, "tcp"=>{"type"=>"long"}, "teredo"=>{"type"=>"long"}, "udp"=>{"type"=>"long"}, "vlan"=>{"type"=>"long"}, "vlan_qinq"=>{"type"=>"long"}}}, "detect"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"alert"=>{"type"=>"long"}}}, "dns"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"memcap_global"=>{"type"=>"long"}, "memcap_state"=>{"type"=>"long"}, "memuse"=>{"type"=>"long"}}}, "file_store"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"open_files"=>{"type"=>"long"}}}, "flow"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"emerg_mode_entered"=>{"type"=>"long"}, "emerg_mode_over"=>{"type"=>"long"}, "icmpv4"=>{"type"=>"long"}, "icmpv6"=>{"type"=>"long"}, "memcap"=>{"type"=>"long"}, "memuse"=>{"type"=>"long"}, "spare"=>{"type"=>"long"}, "tcp"=>{"type"=>"long"}, "tcp_reuse"=>{"type"=>"long"}, "udp"=>{"type"=>"long"}}}, "flow_mgr"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"bypassed_pruned"=>{"type"=>"long"}, "closed_pruned"=>{"type"=>"long"}, "est_pruned"=>{"type"=>"long"}, "flows_checked"=>{"type"=>"long"}, "flows_notimeout"=>{"type"=>"long"}, "flows_removed"=>{"type"=>"long"}, "flows_timeout"=>{"type"=>"long"}, "flows_timeout_inuse"=>{"type"=>"long"}, "new_pruned"=>{"type"=>"long"}, "rows_busy"=>{"type"=>"long"}, "rows_checked"=>{"type"=>"long"}, "rows_empty"=>{"type"=>"long"}, "rows_maxlen"=>{"type"=>"long"}, "rows_skipped"=>{"type"=>"long"}}}, "http"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"memcap"=>{"type"=>"long"}, "memuse"=>{"type"=>"long"}}}, "tcp"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"insert_data_normal_fail"=>{"type"=>"long"}, "insert_data_overlap_fail"=>{"type"=>"long"}, "insert_list_fail"=>{"type"=>"long"}, "invalid_checksum"=>{"type"=>"long"}, "no_flow"=>{"type"=>"long"}, "memuse"=>{"type"=>"long"}, "overlap"=>{"type"=>"long"}, "overlap_diff_data"=>{"type"=>"long"}, "pseudo"=>{"type"=>"long"}, "pseudo_failed"=>{"type"=>"long"}, "reassembly_gap"=>{"type"=>"long"}, "reassembly_memuse"=>{"type"=>"long"}, "rst"=>{"type"=>"long"}, "segment_memcap_drop"=>{"type"=>"long"}, "sessions"=>{"type"=>"long"}, "ssn_memcap_drop"=>{"type"=>"long"}, "stream_depth_reached"=>{"type"=>"long"}, "syn"=>{"type"=>"long"}, "synack"=>{"type"=>"long"}}}}}, "uptime"=>{"type"=>"long"}, "tags"=>{"type"=>"keyword"}}}}}} [2018-12-28T23:14:59,828][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/synlite-suricata_stats-1.0.1 [2018-12-28T23:14:59,913][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//192.168.31.125:9200"]} [2018-12-28T23:14:59,919][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://192.168.31.125:9200/]}} [2018-12-28T23:14:59,920][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://192.168.31.125:9200/, :path=>"/"} [2018-12-28T23:14:59,927][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://192.168.31.125:9200/"} [2018-12-28T23:14:59,957][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6} [2018-12-28T23:14:59,957][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: thetype event field won't be used to determine the document _type {:es_version=>6} [2018-12-28T23:14:59,963][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/etc/logstash/synlite_suricata/templates/synlite_suricata.template.json"} [2018-12-28T23:14:59,978][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"order"=>0, "version"=>10001, "index_patterns"=>"suricata-1.0.1-*", "settings"=>{"index"=>{"number_of_shards"=>3, "number_of_replicas"=>1, "refresh_interval"=>"10s", "codec"=>"best_compression"}}, "mappings"=>{"_default_"=>{"numeric_detection"=>true, "dynamic_templates"=>[{"tcp.ack"=>{"path_match"=>"tcp.ack", "mapping"=>{"type"=>"boolean"}}}, {"tcp.cwr"=>{"path_match"=>"tcp.cwr", "mapping"=>{"type"=>"boolean"}}}, {"tcp.ece"=>{"path_match"=>"tcp.ece", "mapping"=>{"type"=>"boolean"}}}, {"tcp.fin"=>{"path_match"=>"tcp.fin", "mapping"=>{"type"=>"boolean"}}}, {"tcp.psh"=>{"path_match"=>"tcp.psh", "mapping"=>{"type"=>"boolean"}}}, {"tcp.rst"=>{"path_match"=>"tcp.rst", "mapping"=>{"type"=>"boolean"}}}, {"tcp.syn"=>{"path_match"=>"tcp.syn", "mapping"=>{"type"=>"boolean"}}}, {"tcp.urg"=>{"path_match"=>"tcp.urg", "mapping"=>{"type"=>"boolean"}}}, {"string_fields"=>{"match_mapping_type"=>"string", "match"=>"*", "mapping"=>{"type"=>"keyword"}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "alert"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"action"=>{"type"=>"keyword"}, "category"=>{"type"=>"keyword"}, "cve"=>{"type"=>"keyword"}, "gid"=>{"type"=>"long"}, "rev"=>{"type"=>"long"}, "severity"=>{"type"=>"long"}, "signature"=>{"type"=>"keyword"}, "signature_id"=>{"type"=>"long"}}}, "app_proto"=>{"type"=>"keyword"}, "autonomous_system"=>{"type"=>"keyword"}, "city"=>{"type"=>"keyword"}, "client_asn"=>{"type"=>"long"}, "client_autonomous_system"=>{"type"=>"keyword"}, "client_city"=>{"type"=>"keyword"}, "client_country"=>{"type"=>"keyword"}, "client_geo_location"=>{"type"=>"geo_point"}, "client_hostname"=>{"type"=>"keyword"}, "client_ip"=>{"type"=>"ip"}, "country"=>{"type"=>"keyword"}, "dest_asn"=>{"type"=>"long"}, "dest_autonomous_system"=>{"type"=>"keyword"}, "dest_city"=>{"type"=>"keyword"}, "dest_country"=>{"type"=>"keyword"}, "dest_geo_location"=>{"type"=>"geo_point"}, "dest_hostname"=>{"type"=>"keyword"}, "dest_ip"=>{"type"=>"ip"}, "dest_port"=>{"type"=>"long"}, "dest_port_name"=>{"type"=>"keyword"}, "dest_rep_tags"=>{"type"=>"keyword"}, "dns"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"id"=>{"type"=>"long"}, "rcode"=>{"type"=>"keyword"}, "rdata"=>{"type"=>"keyword"}, "rrname"=>{"type"=>"keyword"}, "rrtype"=>{"type"=>"keyword"}, "ttl"=>{"type"=>"long"}, "tx_id"=>{"type"=>"long"}, "type"=>{"type"=>"keyword"}}}, "event"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"host"=>{"type"=>"keyword"}, "subtype"=>{"type"=>"keyword"}, "type"=>{"type"=>"keyword"}}}, "fileinfo"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"filename"=>{"type"=>"keyword"}, "gaps"=>{"type"=>"boolean"}, "size"=>{"type"=>"long"}, "state"=>{"type"=>"keyword"}, "stored"=>{"type"=>"boolean"}, "tx_id"=>{"type"=>"long"}}}, "flow"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"age"=>{"type"=>"long"}, "alerted"=>{"type"=>"boolean"}, "bytes"=>{"type"=>"long"}, "bytes_toclient"=>{"type"=>"long"}, "bytes_toserver"=>{"type"=>"long"}, "end"=>{"type"=>"date"}, "pkts"=>{"type"=>"long"}, "pkts_toclient"=>{"type"=>"long"}, "pkts_toserver"=>{"type"=>"long"}, "reason"=>{"type"=>"keyword"}, "start"=>{"type"=>"date"}, "state"=>{"type"=>"keyword"}}}, "flow_id"=>{"type"=>"long"}, "http"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"hostname"=>{"type"=>"keyword"}, "http_content_type"=>{"type"=>"keyword"}, "http_method"=>{"type"=>"keyword"}, "http_refer"=>{"type"=>"keyword"}, "http_user_agent"=>{"type"=>"keyword"}, "length"=>{"type"=>"long"}, "protocol"=>{"type"=>"keyword"}, "redirect"=>{"type"=>"keyword"}, "status"=>{"type"=>"long"}, "url"=>{"type"=>"keyword"}, "useragent_app"=>{"type"=>"keyword"}, "useragent_app_ver"=>{"type"=>"keyword"}, "useragent_device"=>{"type"=>"keyword"}, "useragent_os"=>{"type"=>"keyword"}, "useragent_os_ver"=>{"type"=>"keyword"}, "xff"=>{"type"=>"keyword"}}}, "icmp_code"=>{"type"=>"long"}, "icmp_type"=>{"type"=>"long"}, "in_iface"=>{"type"=>"keyword"}, "ip_version"=>{"type"=>"keyword"}, "log"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"message"=>{"type"=>"keyword"}, "severity"=>{"type"=>"keyword"}, "tags"=>{"type"=>"keyword"}}}, "node"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ipaddr"=>{"type"=>"ip"}, "hostname"=>{"type"=>"keyword"}}}, "proto"=>{"type"=>"keyword"}, "rep_tags"=>{"type"=>"keyword"}, "server_asn"=>{"type"=>"long"}, "server_autonomous_system"=>{"type"=>"keyword"}, "server_city"=>{"type"=>"keyword"}, "server_country"=>{"type"=>"keyword"}, "server_geo_location"=>{"type"=>"geo_point"}, "server_hostname"=>{"type"=>"keyword"}, "server_ip"=>{"type"=>"ip"}, "service_name"=>{"type"=>"keyword"}, "service_port"=>{"type"=>"long"}, "src_asn"=>{"type"=>"long"}, "src_autonomous_system"=>{"type"=>"keyword"}, "src_city"=>{"type"=>"keyword"}, "src_country"=>{"type"=>"keyword"}, "src_geo_location"=>{"type"=>"geo_point"}, "src_hostname"=>{"type"=>"keyword"}, "src_ip"=>{"type"=>"ip"}, "src_port"=>{"type"=>"long"}, "src_port_name"=>{"type"=>"keyword"}, "src_rep_tags"=>{"type"=>"keyword"}, "tcp"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ecn"=>{"type"=>"boolean"}, "state"=>{"type"=>"keyword"}, "tcp_flags"=>{"type"=>"keyword"}, "tcp_flags_tc"=>{"type"=>"keyword"}, "tcp_flags_ts"=>{"type"=>"keyword"}}}, "tls"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"fingerprint"=>{"type"=>"keyword"}, "issuerdn"=>{"type"=>"keyword"}, "notafter"=>{"type"=>"date"}, "notbefore"=>{"type"=>"date"}, "serial"=>{"type"=>"keyword"}, "session_resumed"=>{"type"=>"boolean"}, "sni"=>{"type"=>"keyword"}, "subject"=>{"type"=>"keyword"}, "version"=>{"type"=>"keyword"}}}, "tx_id"=>{"type"=>"long"}, "vars"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"flowints"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"applayer.anomaly.count"=>{"type"=>"long"}, "http.anomaly.count"=>{"type"=>"long"}, "smtp.anomaly.count"=>{"type"=>"long"}, "tcp.retransmission.count"=>{"type"=>"long"}, "tls.anomaly.count"=>{"type"=>"long"}}}}}, "vlan"=>{"type"=>"long"}, "tags"=>{"type"=>"keyword"}, "traffic_locality"=>{"type"=>"keyword"}}}}}} [2018-12-28T23:14:59,995][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/synlite-suricata-1.0.1 [2018-12-28T23:15:00,055][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//192.168.31.125:9200"]} [2018-12-28T23:15:00,150][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/etc/logstash/synlite_suricata/geoipdbs/GeoLite2-City.mmdb"} [2018-12-28T23:15:00,286][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/etc/logstash/synlite_suricata/geoipdbs/GeoLite2-ASN.mmdb"} [2018-12-28T23:15:22,991][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/etc/logstash/synlite_suricata/geoipdbs/GeoLite2-City.mmdb"} [2018-12-28T23:15:22,993][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/etc/logstash/synlite_suricata/geoipdbs/GeoLite2-ASN.mmdb"}

The thing is when I remove the filter part from conf.d and let only 1 output I get logs. But this seems to be stuck for ever

[robcowart]

How much resources have you given it? What version of Logstash?

[adsanz]

Hi,
I don't what you mean by resources, but if you mean from where are the logs comming from, they are coming from filebeat. The logstash version I used is 6.2.4

[robcowart]

CPU, memory, etc.

[adsanz]

Okey, I had 256M and 1G on jvm.options, I only adjusted elasticsearch did not notice logstash has it too. I adjusted to 2 GB as a test and now it started smoothly and everything seems to work now! Thank you for the help