Type variable conflict
Stybyk opened this issue · 6 comments
After some time of running I´ve got the error:
`
Traceback (most recent call last):
File "./main.py", line 99, in
main()
File "./main.py", line 87, in main
if influxdb_client.write_points(data) == True:
File "/home/pi/.local/lib/python3.7/site-packages/influxdb/client.py", line 599, in write_points
consistency=consistency)
File "/home/pi/.local/lib/python3.7/site-packages/influxdb/client.py", line 676, in _write_points
protocol=protocol
File "/home/pi/.local/lib/python3.7/site-packages/influxdb/client.py", line 410, in write
headers=headers
File "/home/pi/.local/lib/python3.7/site-packages/influxdb/client.py", line 369, in request
raise InfluxDBClientError(err_msg, response.status_code)
influxdb.exceptions.InfluxDBClientError: 400: {"error":"partial write: field type conflict: input field "packetLoss" on measurement "packetLoss" is type float, already exists as type integer dropped=1"}
`
Do you know how to prevent this exception? I can´t imagine why packet loss has resulted in float :D
From the graph, I see that almost all the time is packet loss 0.
But after the script fail I tried speedtest manually:
``
:~ $ speedtest
Speedtest by Ookla
Server: Fione sp. z o.o. - Racibórz (id = 16834)
ISP: RIO Media
Latency: 32.16 ms (62.22 ms jitter)
Download: 93.88 Mbps (data used: 106.5 MB)
Upload: 5.26 Mbps (data used: 9.3 MB)
Packet Loss: 6.5%
``
And I am getting Packet loss 6.5 % , maybe DB considered 0.0 from the first time as int 0.
And now i can´t write new float.
Is it some command to retype this variable type in database ?
Maybe to prevent this bug in future: do python know to force write float ?
I tried a workaround to drop measurement and replace it with float value.
`
DROP MEASUREMENT "packetLoss"
INSERT packetLoss packetLoss=1.2
`
But It is also problem because, when you get packetLose as 0.0 , it is written again as Integer and there is again conflict:
influxdb.exceptions.InfluxDBClientError: 400: {"error":"partial write: field type conflict: input field \"packetLoss\" on measurement \"packetLoss\" is type integer, already exists as type float dropped=1"}
Any idea how to solve it ?
Finally, I did some workaround by adding a small number of packet loss to the measured value,
eg:
'packetLoss': data['packetLoss'] + 0.0001,
For example, 0.0001, is negligible vs real measurement but it will keep function to prevent int creation.
Hey, sorry I never followed up on this issue. My understanding is that the InfluxDB package will derive the database type from the initial Python type so yeah packet loss may be 0 (int) and then 1.2 (float) as you saw.
Just had a look again at the script myself, i've made a few changes below.
import json
import subprocess
from influxdb import InfluxDBClient
# InfluxDB Settings
DB_ADDRESS = 'db_hostname.network'
DB_PORT = 8086
DB_USER = 'db_username'
DB_PASSWORD = 'db_password'
DB_DATABASE = 'speedtest_db'
# Speedtest Settings
TEST_INTERVAL = 1800 # Time between tests (in seconds).
TEST_FAIL_INTERVAL = 60 # Time before retrying a failed Speedtest (in seconds).
influxdb_client = InfluxDBClient(
DB_ADDRESS, DB_PORT, DB_USER, DB_PASSWORD, None)
def init_db():
databases = influxdb_client.get_list_database()
if len(list(filter(lambda x: x['name'] == DB_DATABASE, databases))) == 0:
influxdb_client.create_database(
DB_DATABASE) # Create if does not exist.
else:
influxdb_client.switch_database(DB_DATABASE) # Switch to if does exist.
def format_for_influx(cliout):
data = json.loads(cliout)
# There is additional data in the speedtest-cli output but it is likely not necessary to store.
influx_data = [
{
'measurement': 'ping',
'time': data['timestamp'],
'fields': {
'jitter': float(data['ping']['jitter']),
'latency': float(data['ping']['latency'])
}
},
{
'measurement': 'download',
'time': data['timestamp'],
'fields': {
# Byte to Megabit
'bandwidth': data['download']['bandwidth'] / 125000,
'bytes': data['download']['bytes'],
'elapsed': data['download']['elapsed']
}
},
{
'measurement': 'upload',
'time': data['timestamp'],
'fields': {
# Byte to Megabit
'bandwidth': data['upload']['bandwidth'] / 125000,
'bytes': data['upload']['bytes'],
'elapsed': data['upload']['elapsed']
}
},
{
'measurement': 'packetLoss',
'time': data['timestamp'],
'fields': {
'packetLoss': float(data['packetLoss'])
}
}
]
print(influx_data)
return influx_data
def main():
init_db() # Setup the database if it does not already exist.
while (1): # Run a Speedtest and send the results to influxDB indefinitely.
speedtest = subprocess.run(
["speedtest", "--accept-license", "--accept-gdpr", "-f", "json"], capture_output=True)
if speedtest.returncode == 0: # Speedtest was successful.
data = format_for_influx(speedtest.stdout)
print("Speedtest Successful:")
if influxdb_client.write_points(data) == True:
print("Data written to DB successfully")
time.sleep(TEST_INTERVAL)
else: # Speedtest failed.
print("Speedtest Failed:")
print(speedtest.stderr)
print(speedtest.stdout)
time.sleep(TEST_FAIL_INTERVAL)
if __name__ == '__main__':
print('Speedtest CLI Data Logger to InfluxDB')
main()
I can't actually test the change personally since I don't have an InfluxDB instance set up at the moment. Basically any values that I think may be a float are now type casted to float. So if the value from speedtest is 1, it should insert 1.0 into the DB.
If you don't mind testing this that would be great and i'll commit the changes.
Looks great, write successful into my float defined PacketLoss database.
`
Speedtest CLI Data Logger to InfluxDB
[{'measurement': 'ping', 'time': '2020-10-20T17:14:19Z', 'fields': {'jitter': 0.038, 'latency': 11.561}}, {'measurement': 'download', 'time': '2020-10-20T17:14:19Z', 'fields': {'bandwidth': 93.95044, 'bytes': 93370104, 'elapsed': 8002}}, {'measurement': 'upload', 'time': '2020-10-20T17:14:19Z', 'fields': {'bandwidth': 63.193272, 'bytes': 106022736, 'elapsed': 13205}}, {'measurement': 'packetLoss', 'time': '2020-10-20T17:14:19Z', 'fields': {'packetLoss': 0.0}}]
Speedtest Successful:
Data are written to DB successfully
`
`
show FIELD KEYS
name: download
fieldKey fieldType
bandwidth float
bytes integer
elapsed integer
name: packetLoss
fieldKey fieldType
packetLoss float
name: ping
fieldKey fieldType
jitter float
latency float
name: speedtest
fieldKey fieldType
download float
ping float
upload float
name: upload
fieldKey fieldType
bandwidth float
bytes integer
elapsed integer
``
One more error still crashing the script.
I changed speedtest intervall to 2 minutes to speed up some late eror findings. And usually it stay running up to 2 hours and then it failed with error bellow. Still looks that packetLoss is a little bit problematic value.
``
.....
Info: Data written to DB successfully
Error: Speedtest failed
b''
b'{"error":"Cannot read from socket: Resource temporarily unavailable"}\n'
Info: Speedtest successful
Info: Data written to DB successfully
Info: Speedtest successful
Info: Data written to DB successfully
Info: Speedtest successful
Info: Data written to DB successfully
Info: Speedtest successful
Info: Data written to DB successfully
Info: Speedtest successful
Info: Data written to DB successfully
Traceback (most recent call last):
File "/home/pi/Desktop/main.py", line 136, in
main()
File "/home/pi/Desktop/main.py", line 118, in main
data = format_for_influx(speedtest.stdout)
File "/home/pi/Desktop/main.py", line 87, in format_for_influx
'packetLoss': float(data['packetLoss'])
KeyError: 'packetLoss'
``