UnicodeDecodeError with newer kismet using db version 8 but works with older kismet using db version 6

[SneakyShrike]

I have a python script that pulls out data from a generated kismet file so that the information is presented in a summarised format. This kismet file is generated when you stop running kismet and has an extension of .kismet. This kismet file is actually a SQLite database underneath. However I have an issue whereby when I try to run my python script on a kismet file generated on newer versions of kismet I get quite a few errors, whereas If I run on an older version of kismet I was using before it works fine. I should also point out that I updated all my other libraries and software on my system too. My python script makes use of the kismetdb python wrapper library and a few others to accomplish this extraction of SQLite data.

I have done some investigating and tested a few things. Firstly I checked the version of SQLite that I had used before and what I had now. The one I was using before was version SQLite 3027002 and the one I’m using now is SQLite 3038002. I tested both versions of SQLite with the new version of kismet and it made no difference.
Next I tested different versions of python. My original python version was python 3.7.3 and my new one is now python 3.9.2. I ran my script with both python versions on the new kismet version. On python 3.7.3 I got a different error that says KeyError: 8. Whereas with python 3.9.2 I got a UnicodeDecodeError which I've posted the full error log after my code snippet.

The new kismet version I’m using now is 2022-01-R3 and the one I was using before was 2020-12-R3. Looking at the changelog didn’t really give me an information that I found useful. I also looked at the version of the actual kismet db_versions on both versions of kismet. The newer kismet uses version 8 whereas the older kismet is using version 6 and according to this information version 8 introduces the hash and packetid attributes to the packets table within the generated kismet file.

This is my python script code, I’ve commented out most of it so I can narrow down what’s causing the error on the new kismet version. I’ve managed to determine that it’s when I call the get_all() function that the error occurs. When I run this code with the old kismet version the print(KIS_DEVICES.get_all(**query_args)) outputs all of the devices that kismet detected, which corresponds to the devices table in the generated kismet SQLite file, which is too long to show. But that’s what I’m trying to do but with the new kismet version that uses kismetdb version 8.

import json, sys, kismetdb
from datetime import datetime

# Check if KismetDB is Specified as an Argument
if not len(sys.argv) == 2:
	print("[!] No KismetDB Specified")
	sys.exit(0)

# Set Input and Output Files
KIS_IN = sys.argv[1]
KIS_OUT = "%ssummary" % (KIS_IN[:-6])

query_args = {}
# Get Kismet Devices from DB
KIS_DEVICES = kismetdb.Devices(KIS_IN)
print(KIS_DEVICES.get_all(**query_args))

KIS_DB = [row["device"] for row in KIS_DEVICES.get_all(**query_args)]

sys.exit(0)

This is the error I get when I try to run that same code with the newer kismet version:

sudo python3 KismetDB_to_Summary\ copy.py new\ pi\ build/Kismet-20220411-23-01-06-1.kismet 
Traceback (most recent call last):
  File "/Users/user/Desktop/Apolloo /KismetDB_to_Summary copy.py", line 53, in <module>
    print(KIS_DEVICES.get_all(**query_args))
  File "/usr/local/lib/python3.9/site-packages/kismetdb/base_interface.py", line 155, in get_all
    return self.get_rows(self.column_names, sql, replacements)
  File "/usr/local/lib/python3.9/site-packages/kismetdb/base_interface.py", line 325, in get_rows
    for row in cur.fetchall():
  File "/usr/local/lib/python3.9/site-packages/kismetdb/utility.py", line 473, in device_field_parser
    retval = json.dumps(json.loads(device))
  File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 341, in loads
    s = s.decode(detect_encoding(s), 'surrogatepass')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf5 in position 1286: invalid start byte

I’ve tried looking through the library code that the errors point to but most of it goes over my head and I’m really not sure what could be causing the errors. I have a feeling it’s something to do with the fact that the newer version of kismetdb (version 8) introduces new attributes in the packets table, but I’m not actually sure if that what the issue is.

Any help would be appeciated as I really want to use a newer version of kismet but still be able to extract and filter the generated SQLite data into my summary output.