UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1025-1026: invalid continuation byte

[ch40s]

DEBUG] [2014-12-18 15:56:05.834409] - (ip analytics for 222.88.195.21)
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 238, in run
    self.load_pcap()
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 215, in load_pcap
    self.sniff(stopper=self.stop_sniffing, filter=self.filter, prn=self.handlePacket, stopperTimeout=1, offline=self.engine.setup['SNIFFER_DIR']+"/"+filename)
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 658, in sniff
    r = prn(p)
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 509, in handlePacket
    self.send_flow_statistics(flow)
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 567, in send_flow_statistics
    self.engine.messenger.broadcast(bson_dumps(data), 'sniffer-data', 'flow_statistics_update')
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/bson/json_util.py", line 125, in dumps
    return json.dumps(_json_convert(obj), *args, **kwargs)
  File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1025-1026: invalid continuation byte

[ch40s]

[DEBUG] [2014-12-18 16:01:13.209754] - Fetching session 54936994a1877125b7f4c9ba from memory
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/malcom/Malcom/shmem/SharedData.py", line 29, in __listen_on_channel
    callback(item['data'])
  File "/opt/malcom/Malcom/networking/messenger.py", line 129, in command_handler
    self.publish_to_channel('sniffer-commands', json.dumps(reply))
  File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1025-1026: invalid continuation byte

[tomchop]

Thanks for reporting this. Can you please describe what you were doing? It seems you were uploading a pcap. Can you share it so I can test on my dev instance?

Cheers

[ch40s]

Unfortunately I cannot share that specific pcap file...I will try to reproduce it with a publicly available one if possible.

[ch40s]

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 238, in run
    self.load_pcap()
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 215, in load_pcap
    self.sniff(stopper=self.stop_sniffing, filter=self.filter, prn=self.handlePacket, stopperTimeout=1, offline=self.engine.setup['SNIFFER_DIR']+"/"+filename)
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 658, in sniff
    r = prn(p)
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 509, in handlePacket
    self.send_flow_statistics(flow)
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 563, in send_flow_statistics
    data['flow'] = flow.get_statistics(self.engine.yara_rules)
  File "/opt/malcom/Malcom/networking/flow.py", line 299, in get_statistics
    self.decoded_flow = Decoder.decode_flow(self)
  File "/opt/malcom/Malcom/networking/flow.py", line 23, in decode_flow
    data = Decoder.HTTP_request(flow.payload)
  File "/opt/malcom/Malcom/networking/flow.py", line 97, in HTTP_request
    host = re.search(r'Host: (?P<host>[.\w-]+)(:(?P<port>[\d]{1,5}))?', payload).groupdict()
AttributeError: 'NoneType' object has no attribute 'groupdict'

Got this error while processing this pcap: http://www.snaketrap.co.uk/pcaps/hbot.pcap
It might not be related to the previous, so feel free to track it as a separate issue.

[tomchop]

It is in fact a separate issue. Thanks for reporting it.

[tomchop]

Error does not happen anymore.