Porting to Python3
Opened this issue · 21 comments
Hello everyone.
I'm having some issue with piping the data to semonitor.
My use case is to use a Raspberry Pi to digest the data coming from my FritzBox via its capture interface. The data decoded by semonitor will be pushed to a database (influxSB). Everything should be scripted of course.
These are the pipes I have in mind
fritzdump.sh: from https://github.com/ntop/ntopng/blob/dev/tools/fritzdump.sh it mirrors all the traffic from the port where the inverter is connected in pcap format
tcpdump to filter out only TCP packets of the inverter
tshark to extract data field
unhexlify to convert hex to binary
semonitor with key to decode the messages
[se2influx is my script to upload to the database]
The command line would look like this:
wget -qO- http://$FRITZIP/cgi-bin/capture_notimeout?ifaceorminor=$IFACE&snaplen=&capture=Start&sid=$SID |
tcpdump -r- 'tcp and host 192.168.178.46' -U --immediate-mode -w- |
tshark -i- -T fields -e data |
./utilities/unhexlify.py |
./semonitor.py -k 73110AAB.key -
This pipe chain doesn't work. I've tried also to change few parameters of tcpdump (-w-) or use named pipes.
I've tried to debug this piece per piece. I ran the wget command online from the Terminal and I get:
Then I piped that into tcpdump:
Then I piped that into tshark:
Then I piped to unhexlify:
where I have a very weird behavior.
In this case numbers '36' and '50' are constantly rewritten with new values and in the middle I have an error with utf-8.
BTW, all my system is defaulted to use utf-8, python included.
I'm using Python 3.5
Any help is appreciated.
If you need more info let me know.
Cheers,
Alex
Hi,
Not sure, but you may be making it more complicated than you need to.
I pipe tcpdump straight into semonitor (and then into tee so I can save a copy of the json output, before finally piping it to pickle2graphite.py to send it to a graphite database).
It may be that you need the tshark and unhexlify steps because of the precise details of what the Frtizbox mirrors, (I mirror the traffic using a managed switch, not via a Fritz box), in which case this probably isn't useful to you, so feel free to ignore.
Anyway, just in case it is helpful ....
I've run that successfully (using Python 2.7, on Raspbian Jessie) for quite a while now. I have it all set up to run by invoking a single bash script. It's not totally elegant ( I was still learning bash and scripting when I wrote it) and it's rather tailored to my own set up. Nonetheless a copy is attached as a .txt file just in case it's helpful.
I have also set up a systemd service to start the script as soon as the R Pi boots up. Again, it works but I don't claim it's the prettiest code in the world.
More recently I have been working on a more elegant solution, using 4 separate systemd units, and named pipes on my "development" machine, which runs Raspbian Stretch (the version of systemd in Jessie apparently doesn't support named pipes), which I could also share if you'd like.
Hope that helps, good luck.
PS A couple of the directory names in the script are misleading (eg I put json files in a directory labelled pcapFiles, for historical reasons while I was getting everything working), but I haven't bothered to go back and tidy everything up in case I mess something up :-(
I'm not familiar with fritzbox so I can't comment on anything about that. What semonitor.py expects is stream containing only the solaredge messages in binary. All the ethernet headers, IP headers, TCP headers, TCP handshake messages, and other messages such as ICMP and ARP should be filtered out. If these messages are present, semonitor.py will usually ignore them as @Geoff99 is apparently seeing, but sometimes it gets confused and may skip good data. The deprecated seextract.py was intended to filter out the extra stuff, but it never worked too well, so now a suggestion is to use tshark followed by unhexlify.py. There are probably other utilities that can accomplish the same thing. If you find something better, please share it.
I also use tee to save a copy of the json output while I simultaneously send it to my home automation system.
Thanks for explaining that @jbuehl, and apologies for the misinformation I offered to you, @dragoshenron .
I have been successfully monitoring passively without the tshark and unhexlify steps, but it may just be that I have been (sort of) lucky, because my inverter transmits in the clear, without encryption. I haven't had problems with good data being dropped (or if I have it has been so infrequent that I put it down to a very occasional and momentary bad connection/corrupted transmission)
If you look at the code that reads the messages in passive mode (see se/msg.py, lines 107 and following) semonitor.py loops through the data stream 1 byte at a time, looking ahead for the magic 4 bytes "\x12\x34\x56\x79"that signals the beginning of a SolarEdge message, whereupon it reads the appropriate length message, then recommences looking for the next message (ie the next magic number) again. Unless I am very unlucky, that magic sequence will not appear anywhere in the TCP/IP overhead, so I think (purely by chance) semonitor.py has been filtering out the unnecessary traffic overheads for me :-)
PS my tcpdump step does include a read filter that removes anything that is not a TCP message.
Hi again, @dragoshenron
Just had a quick look at the screen dumps you supplied, and for what it is worth, I looks like everything is working up to and including your tshark step. At least in so far as the messages output (in ASCII) from the tshark step all begin with 12345679, which is the magic number semonitor.py is looking for, in binary.
So I suspect that something is failing in the conversion back from ASCII to binary in the unhexlify.py step. Could be something to do with the utf-8, or could be something entirely different.
Good luck hunting ...
Geoff99, yes it can work without going through the extra filters, but that depends on the traffic on your network. A long SolarEdge data message might be broken up into more then one IP packet and there might even be other traffic that is captured in between those packets, so if semonitor.py is reading that it will see a chunk of garbage in the middle of a message and the message will be discarded.
Admittedly the tcpdump | tshark | unhexlify | semonitor pipeline is a bit cumbersome, but it does seem to work in most situations. If someone can come up with a more elegant solution then I will buy them a beer :)
Ahah, many thanks @jbuehl for the clarification / education. As they say, luck protects the ignorant.
Because of the way I siphon off the passive traffic using a managed switch to mirror only a single port, coupled with the tcpdump filter which only extracts TCP packets involving the fixed (by the DHCP settings in my router) SolarEdge IP address, I'm only vulnerable to the problem of long Solaredge messages broken across more than 1 IP packet; and thus far it seems I've avoided that, purely by chance.
If I get some spare time I may see what I can discover in the tcpdump | tshark | unhexlify area; if I have any success I will report back. (Earning a beer would be nice, but even more gratifying would be making some small contribution to the enormous work that has already been done on this)
(For another project I have recently come across, but have yet to experiment with, a package called pyshark, which seems to invoke tshark filter from inside python; there may be mileage in that direction. I rather think doing anything from first principles about stripping off all the ethernet etc headers would be an enormous and ongoing headache, with all the developments there appear to be in that area to close off assorted security holes that seem to be discovered with depressing regularity)
Back in 2016 when we ran into the encryption issues, I was writing my own little SolarEdge monitoring script. I never bothered replacing my own solution with semonitor, so I'm still using that. I had solved the traffic sniffing issues myself by actually properly parsing the pcap file and reconstructing the TCP streams inside it. It works pretty reliable.
If you would like to go down that road, may find the PCAPParser class in liveupdate.py inspiring:
http://jerweb.nl/downloads/solaredge-logger/
Fwiw, my pipeline is as simple as tcpdump | tee backup.pcap | liveupdate.py. That python script handles everything in my setup.
Thank you very much to everyone for your feedback. Very appreciated!
@jbuehl Thanks for teaching us many things. You deserve a crate of Belgian beer!!!!
@Geoff99 Thanks for review my screencaps and to check that tshark seems working good.
If I open the pcap file coming from the fritzbox with wireshark, I notice many other packets are captured beyond the ones of the inverter (even if on that specific LAN port only the inverter is connected). 'Extra' packets are broadcast, ARP, SSDP, DHCP, IGMP and similar. On the other hand, all the TCP packets captured are related to SE inverter traffic. This is the reason why for you @Geoff99 everything is working good without specific filters.
For sake of completeness, I've tried to feed semonitor directly with the pcap from tcpdump: it exists before any useful data is extracted/printed.
I also had the suspect that unhexlify creates some troubles.
So this is the output of my tshark call:
and I'm not sure the exact format that can be digested by semonitor. Can someone clarify this? Just plain ascii to binary conversion?
I've tried to cat this file to unhexlify but it fails miserably with:
Traceback (most recent call last):
File "./utilities/unhexlify.py", line 16, in
sys.stdout.write(binascii.unhexlify(l.strip()).decode('utf-8'))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe7 in position 4: invalid continuation byte
I will try tomorrow my best to convert the content of my tshark output which is ASCII, to binary.
@Jerrythafast
I downloaded your script. Thanks for sharing! I will have a look at it: I can't use it straight away as I don't have (and don't want to) install an SQL DB. I'll keep you posted.
Have a nice beer everyone ;)
Something funny is going on with your unhexlify.py.
I downloaded your tshark.out.zip, unzipped it, and tried unhexlify on it myself. It is converted to bin format fine for me. I tried it twice in fact, once on a Windows machine, and once on my Raspberry Pi (running Raspbian Stretch). It's perhaps worth noting that in both cases I'm using Python2.7 - I haven't made the switch to Python3 yet (although the unhexlify.py script is so short that I wouldn't really expect it to make much difference -- but I was wrong, see later).
I saved the output of the unhexlify.py on my RPi and fed it to semonitor.py (with the verbose logging option -vvvv), and semonitor.py read it happily enough - just grumbled at me that since it hadn't got the encryption key yet, it was ignoring each message.
As I was typing this, I had another thought (namely Python3 is installed on my RPi) so I went back and tried running unhexlify.py under Python3.5 - and that doesn't work for me! It raises a TypeError, saying the argument of sys.stdout.write() must be str, not bytes.
Not sure I can offer much more help, but at least you know that it's probably something to do with the unhexlify step and the difference between Python3 and Python2.7.
Hi @Geoff99 .
Indeed, I fully confirm this.
cat tshark.out | python2 unhexlify.py works like a charm
cat tshark.out | python3 unhexlify.py gives your same error TypeError: write() argument must be str, not bytes.
FYI, python2 is 2.7.13 and python is 3.5.3. Both running on Raspbian Stretch on a test machine.
On the RPi that should run the semonitor script I have installed only python 3.5.3, so I will try to find a fix to run unhexlilfy on python3 (it shouldn't be so hard and google is my best friend).
@Geoff99 could you please run the following commands and report your outcome?
python2 -c 'import sys; print(sys.getdefaultencoding())' and python3 -c 'import sys; print(sys.getdefaultencoding())' ?
For me python2 gives 'ascii' and python3 gives 'utf-8', as should be by default (side note: for python<2.4 the default enconding was Latin-1 :O ).
Also:
import sys, locale
print(sys.stdout.encoding)
print(locale.getpreferredencoding())
print(sys.getfilesystemencoding())
on my machine is UTF-8 for everything, both python2 and python3.
I'll keep you posted.
Cheers
EDIT:
This is the ouput of print(binascii.unhexlify(l.strip()))
unhexlify.out.zip
So the issue is NOT in binascii but in writing to stdout, right?
I'm also reading here
What a mess they have done...
Looks like you are well on the way.
On my machine python2 is 2.7.13, python3 is 3.5.3
For python2:
sys.getdefaultencoding()) gives ascii
sys.stdout.encoding gives UTF-8
locale.getpreferredencoding() gives UTF-8
sys.getfilesystemencoding() gives UTF-8
For python3:
sys.getdefaultencoding()) gives utf-8
sys.stdout.encoding gives UTF-8
locale.getpreferredencoding() gives UTF-8
sys.getfilesystemencoding() gives UTF-8
best of luck
PS Have you tried the suggestion from the very bottom of the stackoverflow reference you have above, namely
sys.stdout.buffer.write(some_bytes_object)
Hi @dragoshenron , @jbuehl
A quick experiment shows that in unhexlify.py
,
sys.stdout.write(…) works for python2 but not python3, while
sys.stdout.buffer.write(…) works for python3 but not python2.
Not sure how to deal with that in general, but the specific solution in this particular case is fairly straightforward :-)
Great job @Geoff99 !
I made a pull request #102 with your hint.
Now unhexlify should (=will) run smoothly on both python2 and python3.
Not the end of the day yet... :(
The pipeline cat tshark.out | unhexlify | python2 semonitor works like a charm!
Unfortunately the pipeline cat tshark.out | unhexlify | python3 semonitor is broken.
This is the output.
Still issues with this encoding bazar....
Ideas?
Hi @dragoshenron ,
I'm not really up to speed on Python3 (still happily using 2.7) but it certainly looks like the same (how to treat stdin / stdout as byte not str) issue.
The suggested answer here https://stackoverflow.com/a/38939320/4634423 looks promising.
From a quick look at the semonitor code, I think the place to apply the fix so semonitor works for Python3 would be in the openInFile function at line 55 of se/files.py. I haven't tried it - well I started to but found that to test it I'd have to download and install all the Python3 versions of the modules that semonitor needs, which would be a fair bit of fiddling round for me. But you are already set up for semonitor on python3 so, as they say, I'll leave it as an exercise for the reader :-)
Fingers crossed that gets you up and running.
I am impressed how many breaking changes they dare to allow from Python2 to Python3.
However, I edited the file you suggested in this way:
PY3K = sys.version_info >= (3, 0)
def openInFile(inFileName):
if inFileName == "stdin":
if PY3K:
return sys.stdin.buffer
else:
return sys.stdin
else:
# Explicitly specify mode rb to keep windows happy!
return open(inFileName, 'rb')
of course this is working :) and of course gives other errors :( in msg.py [line 115, can't convert 'bytes' object to str implicitly]. Fair enough.
I've tried to do many different things, playing with b"" and .decode() and similar.
Key edit was to change line 93 of msg.py from msg = "" to msg = b"".
Then the script doesn't give any error but it hangs. This is caused - maybe - by a mismatch between 'msg' variable (which became a byte object with the addition of b) and 'magic' variable which is a string.
Changing line 79 of msg.py from magic = "\x12\x34\x56\x79" to magic = b"\x12\x34\x56\x79"
works but then other errors raise in other points as there so many string operation afterwards...
Solution - I think - it is not to ask the script to process bytes object (default stdin type in python3), but rather to convert bytes-object stdin into string.
How? This unknown...
So, freely inspired by this, from the original msg.py I've tried to change only line 110 from nextByte = readBytes(inFile, 1, mode) to nextByte = readBytes(inFile, 1, mode).decode() but gives a 'utf-8' can't decode... error.
BTW,
I'll leave it as an exercise for the reader
is one of the most hated quote from the time I was doing my master ;)
PS: I didn't find specifically hard to install the few modules needed via "pip3 install whatever"
Thoroughly agree with
I am impressed how many breaking changes they dare to allow from Python2 to Python3
I'm starting to recall why I still use python2.7. On the plus side, at least they did increment the major version number (aka signal API changes). ;-)
In the hope I could be a bit more help I made another foray into python3, along the direction you are following, ie convert the byte string (byte array?) into a string. I tend to agree that working through the codebase converting strings to binary would be a big job - after all, most of what semonitor does involves decoding / translating those message bytes into 2 and 4 byte integers, reals, and ASCII characters.
I think the approach you tried on line 110 can never work - the decode must choose some encoding, so if you don't offer one it choose the default. But the underlying solarEdge message genuinely has no (defined) encoding, so whatever encoding you put in there is doomed to fail on some combination of bytes sometime.
I hoped the following might be useful Convert bytes to string but I made negative progress - didn't even get as far as trying it out before I ran into another python3 / python2 difference (python2's iteritems() is gone, replaced by items() in python3).
I've run out of "spare" time for the moment, sorry (other commitments over the next few days). I rather suspect that unless you have some overwhelming reason to stay with python3, the most expedient (and least frustrating) way to get up and running would be to install python2.7 on your intended production machine.
Sorry not to have been help more solving this today :-(
PS: FWIW, another annoying phrase I recall was
it can quite easily be shown that …
PPS: Thanks for the reminder about pip3 install - I began my python "life" on Windows machine at a time when python package installation was a bit less standardised, and under time pressure I tend to forget how much better arranged it all is nowadays, especially in a Linux environment.
I'd just hit comment on the previous message when I realised that the
underlying solarEdge message genuinely has no (defined) encoding, so whatever encoding you put in there is doomed to fail on some combination of bytes sometime
logic probably applies to any attempt to convert the bytes to a python3 string (since any python3 string must have some encoding, even if only implicit). So if you do need to stick with python3, it probably means converting all the existing semonitor constant "character" strings to byte strings, and hoping that nothing else in all the decode machinery that semonitor uses to convert bytes to integer, reals, etc breaks along the way as well.
:-(
@dragoshenron
I should be doing something else, but read up a bit more on python3 bytes vs string instead,
From learning python I see that
The struct Binary-Data Module
Along similar lines, the Python struct module, used to create and extract packed binary data from strings, works in 3.0 as it does in 2.X, but only operates on bytes and bytearray only, not str (which makes sense, given that it's intended for processing binary data, not text):
Since semonitor makes extensive use of struct I think that confirms you'd have to go with python3's bytes type for the data read in from tshark / unhexlify , instead of trying to convert to / work with a python3 str type :-(
Although it's a huge pain to deal with all the incompatibilities, so far pretty much all of the breaking changes in python3 I've met in my life make good sense. Especially the bytes/str and iteritems/xrange stuff mentioned here, those were just design flaws in python2. If you come to think of it, all that semonitor does is handle binary data. There are almost no strings involved in all of this.
There are some helper packages that you could use if you need to support both versions with the same code, such as py23. Those can easily be installed with pip (also on Windows since python2.7). If someday you'd add a setup.py script to semonitor (and perhaps, but not necessarily, upload it to PyPI) you could just pull them in as dependencies when installing, along with eg pyserial.
Thanks @Jerrythafast .
A fair comment and a helpful one. As a general principle I prefer "conceptual" clarity, and the bytes, str change certainly improves that aspect. My current reluctance is pure work avoidance. When I come up for air from other work, I'll look into py23.
Edited in reminder for future reference. It appears that on pypi, the py23 package is called six (because 2*3=6 ….) documentation for py23 = six package
Bye the way, thanks for your earlier link to your PCAPParser class. I've had a look at it, it is quite neat and straightforward, and it's joined my (unfortunately ever growing) list of useful things I'll explore in more depth as soon as I get chance.
@dragoshenron
Sorry to keep spamming you today, but maybe the title of this issue should be amended to mention "using python3" ?
it probably means converting all the existing semonitor constant "character" strings to byte strings
I started to do that way, e.g. with the magic string, but it seems an endless pit of editing
you'd have to go with python3's bytes type for the data read in from tshark / unhexlify
That would be indeed the solution
the most expedient (and least frustrating) way to get up and running would be to install python2.7 on your intended production machine.
This is only a workaround. I'm an engineer, I can't help :)
the title of this issue should be amended
So be it!
@Jerrythafast
I agree that the breaking changes make sense, but
a) it seems than python2 (flawed indeed) was developed by morons
b) there are no major programming language which fixed their mistakes in this way, breaking millions of lines of code.
c) wasn't possible to have a soft transition (for example with deprecated statements) instead to just launch a traceback? When gcc went from version 3 to version 4 I don't remember raising segmentation faults every so often
Thanks for mentioning py23!
Back to the actual semonitor, I'll try to port it to python3 some time (it's almost holiday time...) and in the meantime I'll install and use python2 to run semonitor.
I will also make a PR with se2influxDB.py
Cheers to everyone!