csirtg-smrt error and traceback in 3.0.0a17
Closed this issue · 6 comments
ventz commented
Hi,
Seeing this in 3.0.0a17:
(pasting a few just to see the time pattern, also the process ##)
Mar 4 16:42:06 cif csirtg-smrt[12278]: 2017-03-04 16:42:06,992 - ERROR - csirtg_smrt.smrt[292] - #033[31mlist index out of range#033[0m
Mar 4 16:42:06 cif csirtg-smrt[12278]: 2017-03-04 16:42:06,992 - INFO - csirtg_smrt.smrt[293] - #033[32mskipping: None#033[0m
Mar 4 16:42:06 cif csirtg-smrt[12278]: 2017-03-04 16:42:06,992 - INFO - csirtg_smrt.smrt[102] - #033[32mprocessing /etc/cif/rules/default/alexa.yml.11199.2017-03-04@06:36:56~#033[0m
Mar 4 16:42:06 cif csirtg-smrt[12278]: Process Process-11:
Mar 4 16:42:06 cif csirtg-smrt[12278]: Traceback (most recent call last):
Mar 4 16:42:06 cif csirtg-smrt[12278]: File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Mar 4 16:42:06 cif csirtg-smrt[12278]: self.run()
Mar 4 16:42:06 cif csirtg-smrt[12278]: File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
Mar 4 16:42:06 cif csirtg-smrt[12278]: self._target(*self._args, **self._kwargs)
Mar 4 16:42:06 cif csirtg-smrt[12278]: File "/usr/local/lib/python2.7/dist-packages/csirtg_smrt/smrt.py", line 282, in _run_smrt
Mar 4 16:42:06 cif csirtg-smrt[12278]: for r, f in s.load_feeds(args.rule, feed=args.feed):
Mar 4 16:42:06 cif csirtg-smrt[12278]: File "/usr/local/lib/python2.7/dist-packages/csirtg_smrt/smrt.py", line 103, in load_feeds
Mar 4 16:42:06 cif csirtg-smrt[12278]: r = Rule(path=os.path.join(rule, f))
Mar 4 16:42:06 cif csirtg-smrt[12278]: File "/usr/local/lib/python2.7/dist-packages/csirtg_smrt/rule.py", line 40, in __init__
Mar 4 16:42:06 cif csirtg-smrt[12278]: raise RuleUnsupported('unsupported file type: {}'.format(path))
Mar 4 16:42:06 cif csirtg-smrt[12278]: RuleUnsupported: unsupported file type: /etc/cif/rules/default/alexa.yml.11199.2017-03-04@06:36:56~
and
Mar 4 17:42:06 cif cif-httpd[12254]: 2017-03-04 17:42:06,546 - INFO - werkzeug[87][Thread-206] - #033[32m127.0.0.1 - - [04/Mar/2017 17:42:06] "GET /ping?write=1 HTTP/1.1" 200 -#033[0m
Mar 4 17:42:06 cif csirtg-smrt[12278]: 2017-03-04 17:42:06,547 - INFO - csirtg_smrt.smrt[102] - #033[32mprocessing /etc/cif/rules/default/alexa.yml#033[0m
Mar 4 17:42:06 cif csirtg-smrt[12278]: 2017-03-04 17:42:06,551 - INFO - csirtg_smrt.smrt[283] - #033[32mprocessing: /etc/cif/rules/default - None#033[0m
Mar 4 17:42:06 cif csirtg-smrt[12278]: 2017-03-04 17:42:06,661 - ERROR - csirtg_smrt.smrt[292] - #033[31mlist index out of range#033[0m
Mar 4 17:42:06 cif csirtg-smrt[12278]: 2017-03-04 17:42:06,661 - INFO - csirtg_smrt.smrt[293] - #033[32mskipping: None#033[0m
Mar 4 17:42:06 cif csirtg-smrt[12278]: 2017-03-04 17:42:06,661 - INFO - csirtg_smrt.smrt[102] - #033[32mprocessing /etc/cif/rules/default/alexa.yml.11199.2017-03-04@06:36:56~#033[0m
Mar 4 17:42:06 cif csirtg-smrt[12278]: Process Process-12:
Mar 4 17:42:06 cif csirtg-smrt[12278]: Traceback (most recent call last):
Mar 4 17:42:06 cif csirtg-smrt[12278]: File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Mar 4 17:42:06 cif csirtg-smrt[12278]: self.run()
Mar 4 17:42:06 cif csirtg-smrt[12278]: File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
Mar 4 17:42:06 cif csirtg-smrt[12278]: self._target(*self._args, **self._kwargs)
Mar 4 17:42:06 cif csirtg-smrt[12278]: File "/usr/local/lib/python2.7/dist-packages/csirtg_smrt/smrt.py", line 282, in _run_smrt
Mar 4 17:42:06 cif csirtg-smrt[12278]: for r, f in s.load_feeds(args.rule, feed=args.feed):
Mar 4 17:42:06 cif csirtg-smrt[12278]: File "/usr/local/lib/python2.7/dist-packages/csirtg_smrt/smrt.py", line 103, in load_feeds
Mar 4 17:42:06 cif csirtg-smrt[12278]: r = Rule(path=os.path.join(rule, f))
Mar 4 17:42:06 cif csirtg-smrt[12278]: File "/usr/local/lib/python2.7/dist-packages/csirtg_smrt/rule.py", line 40, in __init__
Mar 4 17:42:06 cif csirtg-smrt[12278]: raise RuleUnsupported('unsupported file type: {}'.format(path))
Mar 4 17:42:06 cif csirtg-smrt[12278]: RuleUnsupported: unsupported file type: /etc/cif/rules/default/alexa.yml.11199.2017-03-04@06:36:56~
and
Mar 4 18:42:06 cif cif-httpd[12254]: 2017-03-04 18:42:06,530 - INFO - werkzeug[87][Thread-208] - #033[32m127.0.0.1 - - [04/Mar/2017 18:42:06] "GET /ping?write=1 HTTP/1.1" 200 -#033[0m
Mar 4 18:42:06 cif csirtg-smrt[12278]: 2017-03-04 18:42:06,532 - INFO - csirtg_smrt.smrt[102] - #033[32mprocessing /etc/cif/rules/default/alexa.yml#033[0m
Mar 4 18:42:06 cif csirtg-smrt[12278]: 2017-03-04 18:42:06,535 - INFO - csirtg_smrt.smrt[283] - #033[32mprocessing: /etc/cif/rules/default - None#033[0m
Mar 4 18:42:06 cif csirtg-smrt[12278]: 2017-03-04 18:42:06,624 - ERROR - csirtg_smrt.smrt[292] - #033[31mlist index out of range#033[0m
Mar 4 18:42:06 cif csirtg-smrt[12278]: 2017-03-04 18:42:06,624 - INFO - csirtg_smrt.smrt[293] - #033[32mskipping: None#033[0m
Mar 4 18:42:06 cif csirtg-smrt[12278]: 2017-03-04 18:42:06,624 - INFO - csirtg_smrt.smrt[102] - #033[32mprocessing /etc/cif/rules/default/alexa.yml.11199.2017-03-04@06:36:56~#033[0m
Mar 4 18:42:06 cif csirtg-smrt[12278]: Process Process-13:
Mar 4 18:42:06 cif csirtg-smrt[12278]: Traceback (most recent call last):
Mar 4 18:42:06 cif csirtg-smrt[12278]: File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Mar 4 18:42:06 cif csirtg-smrt[12278]: self.run()
Mar 4 18:42:06 cif csirtg-smrt[12278]: File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
Mar 4 18:42:06 cif csirtg-smrt[12278]: self._target(*self._args, **self._kwargs)
Mar 4 18:42:06 cif csirtg-smrt[12278]: File "/usr/local/lib/python2.7/dist-packages/csirtg_smrt/smrt.py", line 282, in _run_smrt
Mar 4 18:42:06 cif csirtg-smrt[12278]: for r, f in s.load_feeds(args.rule, feed=args.feed):
Mar 4 18:42:06 cif csirtg-smrt[12278]: File "/usr/local/lib/python2.7/dist-packages/csirtg_smrt/smrt.py", line 103, in load_feeds
Mar 4 18:42:06 cif csirtg-smrt[12278]: r = Rule(path=os.path.join(rule, f))
Mar 4 18:42:06 cif csirtg-smrt[12278]: File "/usr/local/lib/python2.7/dist-packages/csirtg_smrt/rule.py", line 40, in __init__
Mar 4 18:42:06 cif csirtg-smrt[12278]: raise RuleUnsupported('unsupported file type: {}'.format(path))
Mar 4 18:42:06 cif csirtg-smrt[12278]: RuleUnsupported: unsupported file type: /etc/cif/rules/default/alexa.yml.11199.2017-03-04@06:36:56~
wesyoung commented
what does your /etc/cif/rules/default look like; almost looks like somehow default/alexa.yml got .. renamed?
ventz commented
alexa.yml malwaredomains.yml.11025.2017-03-04@06:36:55~
alexa.yml.11199.2017-03-04@06:36:56~ mirc.yml
apwg.yml netlab_360.yml
bambenek.yml nothink.yml
bambenek.yml.11177.2017-03-04@06:36:56~ openbl.yml
blocklist_de.yml openphish.yml
cisco_umbrella.yml packetmail.yml
cisco_umbrella.yml.11083.2017-03-04@06:36:55~ phishtank.yml
csirtg.yml proxyspy_net.yml
danger_rules_sk.yml ransomware_abuse_ch.yml
dataplane.yml sans_edu.yml
dataplane.yml.11329.2017-03-04@06:36:57~ spamhaus.yml
emergingthreats.yml sslbl_abuse_ch.yml
feodotracker.yml vxvault.yml
malc0de.yml zeustracker_abuse_ch.yml
malwaredomains.yml
wesyoung commented
i'm assuming this is your docker instance? kinda wondering if docker is doing something funny with the rules files when you re-install?
looking at the ones that have the timestamp on them, they are the ones i did modify in this release.
ventz commented
Not docker - on AWS :)
I can remove those -- this is after the upgrade btw, (from .16 -> .17) -- so maybe something moved the old rules?
wesyoung commented
ahhhh. i wonder if ansible did that then? yea remove those. i added a ticket to lock smrt down with ".yml$" regex instead of just ".yml"
i've been testing by blowing away the box and re-running from scratch (have some shell scripts to help me out). i don't start testing with an upgrade till we get to the RC stages (too much changes release to release).
wesyoung commented
actually, csirtg-smrt is doing the right thing, we just weren't catching the exception properly:
https://github.com/csirtgadgets/csirtg-smrt-py/compare/fix/180?expand=1
this will still log the error, but it needed a try/except so the process didn't die.
ty!