semonitor.py Output Files
Closed this issue · 3 comments
Llibyddap commented
I think I have everything setup now - but two last questions...
Here is my setup as described in #141.
I have my capture and processing script setup as:
tcpdump -i eth0 -U -w - tcp and host 192.168.2.29| tee capture.pcap | ./solaredge/semonitor.py -d bill2_log -o bill2.json -vvvv
which results in the -d bill2_log output of:
Nov 10 21:48:40 append: False
Nov 10 21:48:40 baudrate: 115200
Nov 10 21:48:40 commands:
Nov 10 21:48:40 datasource: stdin
Nov 10 21:48:40 follow: False
Nov 10 21:48:40 interface: None
Nov 10 21:48:40 keyfile: None
Nov 10 21:48:40 logfile: bill2_log
Nov 10 21:48:40 master: False
Nov 10 21:48:40 outfile: bill2.json
Nov 10 21:48:40 ports: 22222,22221,80
Nov 10 21:48:40 record: None
Nov 10 21:48:40 slaves:
Nov 10 21:48:40 type: None
Nov 10 21:48:40 updatefile: None
Nov 10 21:48:40 verbose: 4
Nov 10 21:48:40 xerror: False
Nov 10 21:48:40 opening stdin
Nov 10 21:48:49 <stdin> --> message: 1 length: 98
Nov 10 21:48:49 data: 12 34 56 79 d4 c3 b2 a1 02 00 04 00 00 00 00 00
Nov 10 21:48:49 data: 00 00 00 00 00 00 04 00 01 00 00 00 41 f6 c8 5d
Nov 10 21:48:49 data: 65 38 0a 00 55 00 00 00 55 00 00 00 00 27 02 16
Nov 10 21:48:49 data: ef 57 dc 7f a4 38 dc fd 08 00 45 00 00 47 c7 76
Nov 10 21:48:49 data: 40 00 30 06 c0 5f 50 ef af 26 c0 a8 02 1d 56 ce
Nov 10 21:48:49 data: 06 61 ac 3a f2 58 00 01 8a 41 50 18 ff 70 c2 05
Nov 10 21:48:49 data: 00 00
Nov 10 21:48:49 <stdin> --> message: 2 length: 101
Nov 10 21:48:49 data: 12 34 56 79 09 00 f6 ff 29 59 fb ff ff ff 57 ef
Nov 10 21:48:49 data: 16 7f 78 03 01 3c 00 00 00 03 00 00 00 9f 2f 41
Nov 10 21:48:49 data: f6 c8 5d 32 4e 0a 00 84 00 00 00 84 00 00 00 dc
Nov 10 21:48:49 data: 7f a4 38 dc fd 00 27 02 16 ef 57 08 00 45 00 00
Nov 10 21:48:49 data: 76 02 db 00 00 40 06 b4 cc c0 a8 02 1d 50 ef af
Nov 10 21:48:49 data: 26 06 61 56 ce 00 01 8a 41 ac 3a f2 77 50 18 08
Nov 10 21:48:49 data: 68 4b 36 00 00
Nov 10 21:48:49 Could not open last0503.msg file, not loading
Nov 10 21:48:49 No data read from last0503.msg. Not loading
Nov 10 21:48:49 dataLen: 0009
Nov 10 21:48:49 dataLenInv: fff6
Nov 10 21:48:49 sequence: 5929
Nov 10 21:48:49 source: fffffffb
Nov 10 21:48:49 dest: 7f16ef57
Nov 10 21:48:49 function: 0378
Nov 10 21:48:49 Discarding 70 extra bytes
Nov 10 21:48:49 data: 41 f6 c8 5d 32 4e 0a 00 84 00 00 00 84 00 00 00
Nov 10 21:48:49 data: dc 7f a4 38 dc fd 00 27 02 16 ef 57 08 00 45 00
Nov 10 21:48:49 data: 00 76 02 db 00 00 40 06 b4 cc c0 a8 02 1d 50 ef
Nov 10 21:48:49 data: af 26 06 61 56 ce 00 01 8a 41 ac 3a f2 77 50 18
Nov 10 21:48:49 data: 08 68 4b 36 00 00
Nov 10 21:48:49 Unknown function 0x0378
Nov 10 21:48:49 <stdin> --> message: 3 length: 148
Nov 10 21:48:49 data: 12 34 56 79 38 00 c7 ff 29 59 57 ef 16 7f fb ff
Nov 10 21:48:49 data: ff ff c2 03 00 00 30 00 41 f6 c8 5d 00 00 00 00
Nov 10 21:48:49 data: 00 00 00 00 00 00 10 c1 00 4b ce c4 00 2b cd 44
Nov 10 21:48:49 data: 00 00 00 00 00 00 00 00 00 00 10 41 00 00 00 00
Nov 10 21:48:49 data: 01 80 67 44 00 a4 03 46 00 00 00 00 ef e3 41 f6
Nov 10 21:48:49 data: c8 5d 2c 57 0a 00 84 00 00 00 84 00 00 00 dc 7f
Nov 10 21:48:49 data: a4 38 dc fd 00 27 02 16 ef 57 08 00 45 00 00 76
Nov 10 21:48:49 data: 02 dc 00 00 40 06 b4 cb c0 a8 02 1d 50 ef af 26
Nov 10 21:48:49 data: 06 61 56 ce 00 01 8a 8f ac 3a f2 77 50 18 08 68
Nov 10 21:48:49 data: 86 f1 00 00
Nov 10 21:48:49 dataLen: 0038
Nov 10 21:48:49 dataLenInv: ffc7
Nov 10 21:48:49 sequence: 5929
Nov 10 21:48:49 source: 7f16ef57
Nov 10 21:48:49 dest: fffffffb
Nov 10 21:48:49 function: 03c2
Nov 10 21:48:49 Discarding 70 extra bytes
Nov 10 21:48:49 data: 41 f6 c8 5d 2c 57 0a 00 84 00 00 00 84 00 00 00
Nov 10 21:48:49 data: dc 7f a4 38 dc fd 00 27 02 16 ef 57 08 00 45 00
Nov 10 21:48:49 data: 00 76 02 dc 00 00 40 06 b4 cb c0 a8 02 1d 50 ef
Nov 10 21:48:49 data: af 26 06 61 56 ce 00 01 8a 8f ac 3a f2 77 50 18
Nov 10 21:48:49 data: 08 68 86 f1 00 00
Nov 10 21:48:49 Unknown function 0x03c2
Nov 10 21:48:50 <stdin> --> message: 4 length: 224
Nov 10 21:48:50 data: 12 34 56 79 38 00 c7 ff 00 00 57 ef 16 7f fe ff
Nov 10 21:48:50 data: ff ff c2 03 00 00 30 00 41 f6 c8 5d 00 00 00 00
Nov 10 21:48:50 data: 00 00 00 00 00 00 10 c1 00 4b ce c4 00 2b cd 44
Nov 10 21:48:50 data: 00 00 00 00 00 00 00 00 00 00 10 41 00 00 00 00
Nov 10 21:48:50 data: 01 80 67 44 00 a4 03 46 00 00 00 00 da 33 41 f6
Nov 10 21:48:50 data: c8 5d ab f5 0c 00 3c 00 00 00 3c 00 00 00 00 27
Nov 10 21:48:50 data: 02 16 ef 57 dc 7f a4 38 dc fd 08 00 45 00 00 28
Nov 10 21:48:50 data: c7 77 40 00 30 06 c0 7d 50 ef af 26 c0 a8 02 1d
Nov 10 21:48:50 data: 56 ce 06 61 ac 3a f2 77 00 01 8a dd 50 10 ff 70
Nov 10 21:48:50 data: 66 c8 00 00 00 00 00 00 00 00 41 f6 c8 5d 93 d3
Nov 10 21:48:50 data: 0d 00 4c 00 00 00 4c 00 00 00 dc 7f a4 38 dc fd
Nov 10 21:48:50 data: 00 27 02 16 ef 57 08 00 45 00 00 3e 02 dd 00 00
Nov 10 21:48:50 data: 40 06 b5 02 c0 a8 02 1d 50 ef af 26 06 61 56 ce
Nov 10 21:48:50 data: 00 01 8a dd ac 3a f2 77 50 18 08 68 50 b7 00 00
Nov 10 21:48:50 dataLen: 0038
I recall reading, but now cannot find it, that semonitor.py will not save encrypted data (or will not process it) to the -o FILE. If that is the case then I may not have my first question. If that is not the case, then I can't figure out why the data is not parsed into the json (e.g., Nov 10 21:48:49 function: 0378). I see that some functions are processed and some are labeled Unknown function. I also see that there are 0503 messages that are not open - i think this is the encryption (which I haven't downloaded the keys yet).
The second question relates to tcpdump vs. tshark. I think I have the tcpdump command appropriately filtered for traffic from the SE Inverter that is TCP, but if not, how would you chain the tcpdump to the tshark to get a filtered stream?
Thank's the the continuing education.
Llibyddap commented
Well - I may have hit enter too fast... After a day - for whatever reason, the first set of data points hit my output file. I'll need to figure out the details... But exciting - assuming I'm not missing something.
I guess my final question is - I didn't capture the encryption key - is there something missing as a result?
{
"batteries_0x0030":{
"7F16EF57":{
"R15563P3SSEG":{
"AlwaysZero_40_float":8425.0,
"AlwaysZero_48_float":NaN,
"AlwaysZero_66_float":-107616408.0,
"AlwaysZero_70_float":6.826565598804779e-41,
"BattCapacityActual":396.5,
"BattCapacityNom":4.5179431944112146e-39,
"BattCharge":0.0,
"BattChargingStatus":19661,
"Date":"2019-11-10",
"EIn":0,
"EOut":0,
"HexConst_52":"ff ff ff ff",
"HexConst_56":"ff ff ff ff",
"Idc":1.0859270332730375e-05,
"Interval":410104,
"Temp":NaN,
"Time":"22:24:48",
"TotalEnergyIn":1176051712,
"TotalEnergyOut":1146945536,
"Undeciphered_data":"2c 01 00 00 00 00 00 00 02 00 00 00 00 00",
"Vdc":6.664491736252387e-10,
"batteryId":"R15563P3SSEG",
"dateTime":1573453488,
"devLen":100,
"devType":"batteries_0x0030",
"seId":"7F16EF57",
"seType":"0x0030"
}
}
},
"events":{
},
"inverters":{
"7F16EF57":{
"Date":"2019-11-10",
"Eac":0.0,
"Eday":28528.193359375,
"Etot":4608790.0,
"Freq":60.01667404174805,
"ID":"7F16EF57",
"Iac":0.9453125,
"Interval":300,
"Pac":0.0,
"Pmax":2288.64111328125,
"Temp":32.924407958984375,
"Time":"22:24:48",
"Uptime":11759,
"Vac":235.203125,
"Vdc":395.75
}
},
"meters_0x0022":{
"7F16EF57":{
"3_Consumption":{
"AlwaysZero_off10_int2":0,
"AlwaysZero_off18_int2":0,
"AlwaysZero_off26_int2":0,
"AlwaysZero_off34_int2":0,
"Date":"2019-11-10",
"E2X":0,
"EfromX":196,
"Flag_off12_hex":"00 80",
"Flag_off20_hex":"00 80",
"Flag_off28_hex":"00 80",
"Flag_off36_hex":"00 80",
"Interval":300,
"P2X":NaN,
"PfromX":2382.5517578125,
"Time":"22:24:48",
"TotalE2Grid":0,
"TotalEfromGrid":0,
"Totaloff22_int4":0,
"Totaloff30_int4":0,
"dateTime":1573453488,
"devLen":58,
"devType":"meters_0x0022",
"onlyIntervalData":1,
"recType":3,
"seId":"7F16EF57",
"seType":"0x0022"
},
"5_GridImportExport":{
"AlwaysZero_off10_int2":0,
"AlwaysZero_off18_int2":0,
"AlwaysZero_off26_int2":0,
"AlwaysZero_off34_int2":0,
"Date":"2019-11-10",
"E2X":0,
"EfromX":196,
"Flag_off12_hex":"00 00",
"Flag_off20_hex":"00 00",
"Flag_off28_hex":"00 00",
"Flag_off36_hex":"00 00",
"Interval":300,
"P2X":0.0,
"PfromX":2388.5517578125,
"Time":"22:24:48",
"TotalE2Grid":15976987,
"TotalEfromGrid":7212562,
"Totaloff22_int4":677578,
"Totaloff30_int4":27532337,
"dateTime":1573453488,
"devLen":58,
"devType":"meters_0x0022",
"onlyIntervalData":0,
"recType":5,
"seId":"7F16EF57",
"seType":"0x0022"
},
"6_UnrecognisedRecType":{
"AlwaysZero_off10_int2":0,
"AlwaysZero_off18_int2":0,
"AlwaysZero_off26_int2":0,
"AlwaysZero_off34_int2":0,
"Date":"2019-11-10",
"E2X":0,
"EfromX":0,
"Flag_off12_hex":"00 80",
"Flag_off20_hex":"00 80",
"Flag_off28_hex":"00 80",
"Flag_off36_hex":"00 80",
"Interval":300,
"P2X":NaN,
"PfromX":0.0,
"Time":"22:24:48",
"TotalE2Grid":0,
"TotalEfromGrid":0,
"Totaloff22_int4":0,
"Totaloff30_int4":0,
"dateTime":1573453488,
"devLen":58,
"devType":"meters_0x0022",
"onlyIntervalData":1,
"recType":6,
"seId":"7F16EF57",
"seType":"0x0022"
}
}
},
"optimizers":{
}
}{
"events":{
},
"inverters":{
},
"meters_0x0022":{
"7F16EF57":{
"7_Battery":{
"AlwaysZero_off10_int2":0,
"AlwaysZero_off18_int2":0,
"AlwaysZero_off26_int2":0,
"AlwaysZero_off34_int2":0,
"Date":"2019-11-10",
"E2X":0,
"EfromX":0,
"Flag_off12_hex":"00 80",
"Flag_off20_hex":"00 80",
"Flag_off28_hex":"00 80",
"Flag_off36_hex":"00 80",
"Interval":300,
"P2X":0.0,
"PfromX":NaN,
"Time":"22:24:48",
"TotalE2Grid":0,
"TotalEfromGrid":0,
"Totaloff22_int4":0,
"Totaloff30_int4":0,
"dateTime":1573453488,
"devLen":58,
"devType":"meters_0x0022",
"onlyIntervalData":1,
"recType":7,
"seId":"7F16EF57",
"seType":"0x0022"
},
"8_MostlyZeroes":{
"AlwaysZero_off10_int2":0,
"AlwaysZero_off18_int2":0,
"AlwaysZero_off26_int2":0,
"AlwaysZero_off34_int2":0,
"Date":"2019-11-10",
"E2X":0,
"EfromX":0,
"Flag_off12_hex":"00 80",
"Flag_off20_hex":"00 80",
"Flag_off28_hex":"00 80",
"Flag_off36_hex":"00 80",
"Interval":300,
"P2X":0.0,
"PfromX":NaN,
"Time":"22:24:48",
"TotalE2Grid":0,
"TotalEfromGrid":0,
"Totaloff22_int4":0,
"Totaloff30_int4":0,
"dateTime":1573453488,
"devLen":58,
"devType":"meters_0x0022",
"onlyIntervalData":1,
"recType":8,
"seId":"7F16EF57",
"seType":"0x0022"
},
"9_PVProduction":{
"AlwaysZero_off10_int2":0,
"AlwaysZero_off18_int2":0,
"AlwaysZero_off26_int2":0,
"AlwaysZero_off34_int2":0,
"Date":"2019-11-10",
"E2X":0,
"EfromX":0,
"Flag_off12_hex":"00 80",
"Flag_off20_hex":"00 80",
"Flag_off28_hex":"00 80",
"Flag_off36_hex":"00 80",
"Interval":300,
"P2X":0.0,
"PfromX":NaN,
"Time":"22:24:48",
"TotalE2Grid":0,
"TotalEfromGrid":0,
"Totaloff22_int4":0,
"Totaloff30_int4":0,
"dateTime":1573453488,
"devLen":58,
"devType":"meters_0x0022",
"onlyIntervalData":1,
"recType":9,
"seId":"7F16EF57",
"seType":"0x0022"
}
}
},
"optimizers":{
}
}
jbuehl commented
If you are getting JSON in the output file and you haven't captured the encryption key, then your inverter must be sending the data unencrypted. The "unknown function" and "discarding extra bytes" messages are caused by extra junk in the captured data that semonitor.py can't interpret. You should filter them out with tshark like this:
tcpdump ... | tee ... | tshark ... | xxd ... | semonitor ...
Llibyddap commented
Adding the extra filters did reduce the junk. Thanks for the confirmation.