Errors - Ubuntu 14.04
Closed this issue · 5 comments
I am on Ubuntu server 14.04 LTS VM with python 2.7.6 installed. I needed to install libssl-dev and libffi-dev using apt-get in order pyopenssl and all the rest to be installed without errors. I also installed service_identity using pip to avoid a warning when running malcom. I think I don't forget something else... :)
I guess something is still missing because I get the following when I run malcom:
./malcom.py -a
....
[DEBUG] - Could not send message: 'NoneType' object has no attribute 'send'
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
./malcom.py -f
...
[DEBUG] - Starting thread for feed TorExitNodes...
[DEBUG] - Could not send message: 'NoneType' object has no attribute 'send'
Exception in thread Thread-5:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(_self.__args, *_self.__kwargs)
File "/opt/malcom/Malcom/feeds/feed.py", line 62, in run
status = self.update()
File "/opt/malcom/Malcom/feeds/dshield_as16276.py", line 23, in update
self.analyze(line)
File "/opt/malcom/Malcom/feeds/dshield_as16276.py", line 40, in analyze
ip, status = self.analytics.save_element(ip, with_status=True)
File "/opt/malcom/Malcom/analytics/analytics.py", line 81, in save_element
return self.data.save(element, with_status=with_status)
File "/opt/malcom/Malcom/model/model.py", line 123, in save
status = self.elements.update({'value': element['value']}, {"$set" : element, "$addToSet": {'tags' : {'$each': tags}}}, upsert=True)
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/pymongo/collection.py", line 561, in update
check_keys, self.uuid_subtype), safe)
InvalidDocument: Cannot encode object: {'refresh_period': 259200, 'type': 'ip', 'value': '178.32.230.51'}
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
When trying to delete a session:
Traceback (most recent call last):
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/gevent/pywsgi.py", line 508, in handle_one_response
self.run_application()
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/geventwebsocket/handler.py", line 88, in run_application
return super(WebSocketHandler, self).run_application()
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/gevent/pywsgi.py", line 494, in run_application
self.result = self.application(self.environ, self.start_response)
File "/opt/malcom/Malcom/web/webserver.py", line 76, in malcom_app
return app(environ, start_response)
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1836, in call
return self.wsgi_app(environ, start_response)
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/opt/malcom/Malcom/web/webserver.py", line 448, in sniffer_session_delete
g.a.data.del_sniffer_session(session_name)
File "/opt/malcom/Malcom/model/model.py", line 99, in del_sniffer_session
filename = session['name'] + ".pcap"
TypeError: 'NoneType' object has no attribute 'getitem'
Thanks for the bug report! The Cannot encode object error comes from a weird incompatibility issue with a certain version of pymongo (the latest one I think) and mongodb. To fix it, roll back to pymongo version 2.4.1. You can do that using pip install pymongo==2.4.1
I don't know where the session deletion bug comes from. I've refactored a lot of code recently (these last two weeks) and now it all runs smoothly on Ubuntu 14.04; tested by copying-pasting commands from a fresh install (I updated the dev. README accordingly). Out of curiosity, which commit is giving you the errors?
Thanks for the service_identity warnings, I noticed them too. I'll incorporate the changes and test them before I merge everything into master.
Cheers!
Rolling back to pymongo version 2.4.1 actually worked. Session deletion also works now.
The only issue I am currently facing is that the feeds are not being updated.
When I press the "Run now" button:
"[DEBUG] - Could not send message: Socket is dead
[DEBUG] - Call to analytics API
[DEBUG] - Got websocket
[DEBUG] - Received: {u'cmd': u'analyticsstatus'}"
After I while on the same page I see:
Status: "ERROR: <urlopen error [Errno 110] Connection timed out>"
When I run malcom.py with "-f" I get:
Starting feed scheduler...
[DEBUG] - Checking feeds...
...
[DEBUG] - Starting thread for feed DShieldSuspiciousDomainsMedium...
[DEBUG] - Could not send message: 'NoneType' object has no attribute 'send'
[DEBUG] - Starting thread for feed DShield16276...
[DEBUG] - Could not send message: 'NoneType' object has no attribute 'send'
...
Could be because of the proxy settings, so which is the recommended way to define them? Export http_proxy and https_proxy didn't help, I have no issues with wget, apt-get etc.
I also get many errors like this while a sniffing session is running or a pcap being played:
"[DEBUG] - Could not send flow statistics: Socket is dead"
Also got this during "Play PCAP":
"[ERROR] - Could not get IP info for A.B.C.D: 'NoneType' object is not iterable"
By the way, I am on the latest commit of the dev branch.
The feeds issue is definitely related to the proxy settings. I have had the same problem and during troubleshooting the issue found that the server that I am running the Malcom instance on is attempting to DNS query and send the feed request out from the server rather than via the proxy, which I have set for the system. Have not found a way as of yet to force Malcom to use a proxy and had to open the system up so that it could query feeds without it.
According to the official Python doc, urllib2.urlopen (the method used for requesting feeds) takes care of proxies transparently:
In addition, if proxy settings are detected (for example, when a *_proxy environment variable like http_proxy is set), ProxyHandler is default installed and makes sure the requests are handled through the proxy.
If you're running Malcom as root, then maybe the problem is that the environment variables are set in your user environment and not root's.
As for the errors "[DEBUG] - Could not send flow statistics: Socket is dead", this is because Malcom can't find the websocket to send information to. Are the nodes popping up on your screen and are the flows updating as they should?
If you're running Malcom behind a reverse proxy, you should take a look at the file webserver.py, right before the first functions are defined. There's a commented block giving settings to use for when the Malcom instance is behind a reverse proxy (like nginx).
Let me know if that helped. Also, please give me the exact commit you're working on since a lot has changed in the dev version in the past few days. I'm still developing a couple of features I'm planning to release (and merge into the master branch) this weekend.
Cheers!
I set the environment variables again just before running malcom.py and the feeds are being updated. :)
In regards to the other error, yes the nodes are popping up and the flows are being updated. I am not using a reverse proxy.
Commit: 8ccf568
I don't get many errors anymore, so I will close this issue as resolved. Thanks a lot for the feedback.