Module Not Found Error
Closed this issue · 28 comments
Hello!
Hopefully a not-too-dumb question, but when trying to run main.py I get the following error:
pi@raspberrypi:~/speedtest-to-influxdb $ python3 ./main.py
Traceback (most recent call last):
File "./main.py", line 5, in <module>
from influxdb import InfluxDBClient
ModuleNotFoundError: No module named 'influxdb'
I have installed the Python InfluxdDB module with # sudo pip install influxdb and confirmed it $ python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())" | xargs ls which does return influxdb and influxdb-3.0.0.egg-info_
My only (beginner level) guess is that the path to the module isn't working? Any ideas that might get me pointed in the right direction?
Hi, definitely not too dumb.
Could it be something to do with pip vs pip3?
Try installing python3-pip, and installing the package with pip3 install influxdb. See how that goes.
Hey! Glad to know it's not too idiotic. We're making progress - I've installed python3:
pi@raspberrypi:~/Python-3.6.3 $ /usr/bin/python3 -V
Python 3.7.3
and installed influxdb as sudo (got a permissions issue otherwise).
Now when running main.py I get the following:
pi@raspberrypi:~/speedtest-to-influxdb $ python3 ./main.py
Speedtest CLI Data Logger to InfluxDB
Traceback (most recent call last):
File "./main.py", line 98, in <module>
main()
File "./main.py", line 81, in main
["speedtest", "--accept-license", "--accept-gdpr", "-f", "json"], capture_output=True)
File "/usr/local/lib/python3.6/subprocess.py", line 403, in run
with Popen(*popenargs, **kwargs) as process:
TypeError: __init__() got an unexpected keyword argument 'capture_output'
Any further ideas? I'm keeping my fingers crossed, this seems like a fantastic opportunity for me to learn a bit more about influxdb and grafana! Thanks again!
After a quick visit to stackoverflow it would seem that this is a difference with Python 3.6 and Python 3.7+ which my system and the Docker container uses.
Try adding the line from subprocess import PIPE below import subprocess and replace capture_output=True with stdout=PIPE, stderr=PIPE
Or you can just replace the contents of main.py with that below. Remembering to adjust the configuration variables accordingly obviously. Should work after that.
import time
import json
import subprocess
from subprocess import PIPE
from influxdb import InfluxDBClient
# InfluxDB Settings
DB_ADDRESS = 'db_hostname.network'
DB_PORT = 8086
DB_USER = 'db_username'
DB_PASSWORD = 'db_password'
DB_DATABASE = 'speedtest_db'
# Speedtest Settings
TEST_INTERVAL = 1800 # Time between tests (in seconds).
TEST_FAIL_INTERVAL = 60 # Time before retrying a failed Speedtest (in seconds).
influxdb_client = InfluxDBClient(
DB_ADDRESS, DB_PORT, DB_USER, DB_PASSWORD, None)
def init_db():
databases = influxdb_client.get_list_database()
if len(list(filter(lambda x: x['name'] == DB_DATABASE, databases))) == 0:
influxdb_client.create_database(
DB_DATABASE) # Create if does not exist.
else:
influxdb_client.switch_database(DB_DATABASE) # Switch to if does exist.
def format_for_influx(cliout):
data = json.loads(cliout)
# There is additional data in the speedtest-cli output but it is likely not necessary to store.
influx_data = [
{
'measurement': 'ping',
'time': data['timestamp'],
'fields': {
'jitter': data['ping']['jitter'],
'latency': data['ping']['latency']
}
},
{
'measurement': 'download',
'time': data['timestamp'],
'fields': {
# Byte to Megabit
'bandwidth': data['download']['bandwidth'] / 125000,
'bytes': data['download']['bytes'],
'elapsed': data['download']['elapsed']
}
},
{
'measurement': 'upload',
'time': data['timestamp'],
'fields': {
# Byte to Megabit
'bandwidth': data['upload']['bandwidth'] / 125000,
'bytes': data['upload']['bytes'],
'elapsed': data['upload']['elapsed']
}
},
{
'measurement': 'packetLoss',
'time': data['timestamp'],
'fields': {
'packetLoss': data['packetLoss']
}
}
]
return influx_data
def main():
init_db() # Setup the database if it does not already exist.
while (1): # Run a Speedtest and send the results to influxDB indefinitely.
speedtest = subprocess.run(
["speedtest", "--accept-license", "--accept-gdpr", "-f", "json"], stdout=PIPE, stderr=PIPE)
if speedtest.returncode == 0: # Speedtest was successful.
data = format_for_influx(speedtest.stdout)
print("Speedtest Successful:")
if influxdb_client.write_points(data) == True:
print("Data written to DB successfully")
time.sleep(TEST_INTERVAL)
else: # Speedtest failed.
print("Speedtest Failed:")
print(speedtest.stderr)
print(speedtest.stdout)
time.sleep(TEST_FAIL_INTERVAL)
if __name__ == '__main__':
print('Speedtest CLI Data Logger to InfluxDB')
main()Thanks so much for the fix dude! I think that moved things forward... but still not out of the woods entirely. After making the above fixes, here's the result:
pi@raspberrypi:~/speedtest-to-influxdb $ python3 ./main.py
Speedtest CLI Data Logger to InfluxDB
Traceback (most recent call last):
File "./main.py", line 99, in <module>
main()
File "./main.py", line 85, in main
data = format_for_influx(speedtest.stdout)
File "./main.py", line 34, in format_for_influx
data = json.loads(cliout)
File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Could you post the output of speedtest --accept-license --accept-gdpr -f json?
You will need to wait a bit as there is no progress indicator with these settings.
See if you get a valid output.
Interestingly enough the output popped right up!
pi@raspberrypi:~/speedtest-to-influxdb $ speedtest --accept-license --accept-gdpr -f json
usage: speedtest [-h] [--no-download] [--no-upload] [--single] [--bytes]
[--share] [--simple] [--csv] [--csv-delimiter CSV_DELIMITER]
[--csv-header] [--json] [--list] [--server SERVER]
[--exclude EXCLUDE] [--mini MINI] [--source SOURCE]
[--timeout TIMEOUT] [--secure] [--no-pre-allocate]
[--version]
speedtest: error: unrecognized arguments: --accept-license --accept-gdpr -f json
Maybe try running speedtest with no arguments, see if that works. If it does try -f json , accept-gdpr etc, and see which is unrecognised?
Hmm speedtest works just fine, but none of the other commands are recognised, oddly enough...
pi@raspberrypi:~/speedtest-to-influxdb $ -f json
-bash: -f: command not found
pi@raspberrypi:~/speedtest-to-influxdb $ --accept-gdpr
-bash: --accept-gdpr: command not found
pi@raspberrypi:~/speedtest-to-influxdb $ --accept-license
-bash: --accept-license: command not found
pi@raspberrypi:~/speedtest-to-influxdb $
You need to run speedtest and pass -f json, --accept-gdpr and --accept-license as arguments.
Such as,
speedtest -f json
speedtest -f json --accept-license --accept-gdpr
Ah well that was rather daft of me! Here's what we got:
pi@raspberrypi:~/speedtest-to-influxdb $ speedtest -f json
usage: speedtest [-h] [--no-download] [--no-upload] [--single] [--bytes]
[--share] [--simple] [--csv] [--csv-delimiter CSV_DELIMITER]
[--csv-header] [--json] [--list] [--server SERVER]
[--exclude EXCLUDE] [--mini MINI] [--source SOURCE]
[--timeout TIMEOUT] [--secure] [--no-pre-allocate]
[--version]
speedtest: error: unrecognized arguments: -f json
pi@raspberrypi:~/speedtest-to-influxdb $ speedtest -f json --accept-license --accept-gdpr
usage: speedtest [-h] [--no-download] [--no-upload] [--single] [--bytes]
[--share] [--simple] [--csv] [--csv-delimiter CSV_DELIMITER]
[--csv-header] [--json] [--list] [--server SERVER]
[--exclude EXCLUDE] [--mini MINI] [--source SOURCE]
[--timeout TIMEOUT] [--secure] [--no-pre-allocate]
[--version]
speedtest: error: unrecognized arguments: -f json --accept-license --accept-gdpr
pi@raspberrypi:~/speedtest-to-influxdb $ speedtest --accept-license --accept-gdpr
usage: speedtest [-h] [--no-download] [--no-upload] [--single] [--bytes]
[--share] [--simple] [--csv] [--csv-delimiter CSV_DELIMITER]
[--csv-header] [--json] [--list] [--server SERVER]
[--exclude EXCLUDE] [--mini MINI] [--source SOURCE]
[--timeout TIMEOUT] [--secure] [--no-pre-allocate]
[--version]
speedtest: error: unrecognized arguments: --accept-license --accept-gdpr
Does speedtest run without any arguments at all?
Perfectly:
pi@raspberrypi:~/speedtest-to-influxdb $ speedtest
Retrieving speedtest.net configuration...
Testing from BT (86.184.77.75)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Redraw Internet (London) [5.22 km]: 18.629 ms
Testing download speed...............................................................................
.Download: 33.83 Mbit/s
Testing upload speed................................................................................................
Upload: 16.80 Mbit/s
I really wish I had any ideas here... and really do appreciate the help on this!
Should have noticed this before, it seems you have installed speedtest-cli from the Debian repositories. It is an unofficial client which is launched with the same command (speedtest).
This script is for use with the relatively recently released official client from Ookla. As their client is proprietary it can't be included in the Debian repositories.
You should be able to get this working by doing the following.
-
Uninstall speedtest-cli
sudo apt remove speedtest-cli -
Install the Ooka Speedtest application (https://www.speedtest.net/apps/cli)
sudo apt-get install gnupg1 apt-transport-https dirmngrexport INSTALL_KEY=379CE192D401AB61export DEB_DISTRO=$(lsb_release -sc)sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys $INSTALL_KEYecho "deb https://ookla.bintray.com/debian ${DEB_DISTRO} main" | sudo tee /etc/apt/sources.list.d/speedtest.listsudo apt updatesudo apt install speedtest
Try running the script then, should all work.
Ah super useful notice! So, before running through all of that, I attempted to uninstall the incorrect speedtest and got the following:
pi@raspberrypi:~/speedtest-to-influxdb $ sudo apt remove speedtest-cli
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package 'speedtest-cli' is not installed, so not removed
The following package was automatically installed and is no longer required:
point-rpi
Use 'sudo apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 12 not upgraded.
Before attempting to install the right app, I just wanted to check that we weren't missing something as it doesn't look like speedtest-cli was installed. I tried uninstalling both at ~ as well as ~/speedtest-to-influxdb
Right, so it looks like speedtest-cli didn't uninstall correctly.
What does which speedtest output?
Always something! Here's the result:
pi@raspberrypi:~/speedtest-to-influxdb $ which speedtest
/usr/local/bin/speedtest
So if you run /usr/local/bin/speedtest --version what do you get?
You're absolutely right on the flavor of speedtest!
pi@raspberrypi:~/speedtest-to-influxdb $ /usr/local/bin/speedtest --version
speedtest-cli 2.1.2
Python 2.7.16 (default, Oct 10 2019, 22:02:15) [GCC 8.3.0]
Okay, so you'll have to manually remove it which isn't ideal but should work for this.
sudo rm /usr/local/bin/speedtest
Then try installing the Ookla Speedtest with the instructions in step 2 above (or on the Ookla website).
At long last, seeming success! Thanks so much mate!
pi@raspberrypi:~/speedtest-to-influxdb $ python3 ./main.py
Speedtest CLI Data Logger to InfluxDB
Speedtest Successful:
Data written to DB successfully
Question - it seems that the script is actively running and I'm guessing I'm not supposed to exit out of it? Therefore should I run it in a screen or something to allow it to do it's thing? Additionally as it does it's own loop, I don't need to set it up as a cronjob or anything right?
Yeah, quitting the script will kill it so you will have to find your own way of starting it on boot.
Without modifying the script, i'd probably make it a systemd service which starts on boot and leaves it running in the background. You can change the frequency that it runs a speedtest with a variable in the script.
This tutorial looks promising.
Amazing, thanks again so much for everything! Now on to figuring out Grafana!
No worries, glad to help.
As for running the script if you would prefer to use cron to schedule tests you could replace time.sleep(TEST_INTERVAL) with quit() and the script will stop after a successful test.
Ah thanks! I'll have a think about whether or not cron or screen/systemd might be best! I hate to come back with a other error so quickly, but I rebooted the pi just to try and recreate everything from a clean slate - this is what happened:
pi@raspberrypi:~/speedtest-to-influxdb $ python3 ./main.py
Speedtest CLI Data Logger to InfluxDB
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/local/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 184, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0xb5d3b830>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8086): Max retries exceeded with url: /query?q=SHOW+DATABASES (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xb5d3b830>: Failed to establish a new connection: [Errno 111] Connection refused',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./main.py", line 99, in <module>
main()
File "./main.py", line 78, in main
init_db() # Setup the database if it does not already exist.
File "./main.py", line 24, in init_db
databases = influxdb_client.get_list_database()
File "/usr/local/lib/python3.6/site-packages/influxdb/client.py", line 617, in get_list_database
return list(self.query("SHOW DATABASES").get_points())
File "/usr/local/lib/python3.6/site-packages/influxdb/client.py", line 450, in query
expected_response_code=expected_response_code
File "/usr/local/lib/python3.6/site-packages/influxdb/client.py", line 283, in request
timeout=self._timeout
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8086): Max retries exceeded with url: /query?q=SHOW+DATABASES (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xb5d3b830>: Failed to establish a new connection: [Errno 111] Connection refused',))
... I just have my head in my hands at this point...
Seems like InfluxDB might not be running.
What does systemctl status influxdb say?
Ah it returns this:
pi@raspberrypi:~/speedtest-to-influxdb $ systemctl status influxdb
● influxdb.service - InfluxDB is an open-source, distributed, time series database
Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-01-09 16:00:00 GMT; 11min ago
Docs: https://docs.influxdata.com/influxdb/
Main PID: 242 (influxd)
Memory: 186.7M
CGroup: /system.slice/influxdb.service
└─242 /usr/bin/influxd -config /etc/influxdb/influxdb.conf
Jan 09 16:06:17 raspberrypi influxd[242]: ts=2020-01-09T16:06:17.379755Z lvl=info msg="Starting snapshot service
Jan 09 16:06:17 raspberrypi influxd[242]: ts=2020-01-09T16:06:17.385005Z lvl=info msg="Starting continuous query
Jan 09 16:06:17 raspberrypi influxd[242]: ts=2020-01-09T16:06:17.389076Z lvl=info msg="Starting HTTP service" lo
Jan 09 16:06:17 raspberrypi influxd[242]: ts=2020-01-09T16:06:17.389405Z lvl=info msg="opened HTTP access log" l
Jan 09 16:06:17 raspberrypi influxd[242]: ts=2020-01-09T16:06:17.389586Z lvl=info msg="Auth is enabled but share
Jan 09 16:06:17 raspberrypi influxd[242]: ts=2020-01-09T16:06:17.391671Z lvl=info msg="Listening on HTTP" log_id
Jan 09 16:06:17 raspberrypi influxd[242]: ts=2020-01-09T16:06:17.397245Z lvl=info msg="Starting retention policy
Jan 09 16:06:17 raspberrypi influxd[242]: ts=2020-01-09T16:06:17.399308Z lvl=info msg="Listening for signals" lo
Jan 09 16:06:17 raspberrypi influxd[242]: ts=2020-01-09T16:06:17.400008Z lvl=info msg="Storing statistics" log_i
Jan 09 16:06:18 raspberrypi influxd[242]: ts=2020-01-09T16:06:18.206637Z lvl=info msg="Sending usage statistics
lines 1-19/19 (END)
[1]+ Stopped systemctl status influxdb
Okay, so it's running. What's the output of sudo netstat -ltnp? If this doesn't run you may need to run sudo apt install net-tools.
Ah huh, it worked when I ran it again! It must be that influxdb is just slow to boot up when restarting. I noticed that yesterday when running influx it would take a good 3-4 minutes until I got the console line. I kinda presumed that was because I was running all this off an ancient Raspi3 that I had in the closet, but thats only a finger in the air assumption. When I ran main.py again I got what we were looking for:
pi@raspberrypi:~/speedtest-to-influxdb $ python3 ./main.py
Speedtest CLI Data Logger to InfluxDB
Speedtest Successful:
Data written to DB successfully
^Z
[2]+ Stopped python3 ./main.py