Would it be worth adding an option for the newer `influxdb-client` library?
signebedi opened this issue · 5 comments
First off, this is a great script. Have been using it for a bit now and am very impressed with how well written it is. I've noticed that it's using the influxdb pip library, which was archived last year. The new library is called influxdb-client, which is actively maintained. This might be a response to major API changes in v2 which adds Flux queries. The new library uses a slightly different convention for making a database connection - here's the sample from the getting started guide:
import influxdb_client
from influxdb_client.client.write_api import SYNCHRONOUS
bucket = "<my-bucket>"
org = "<my-org>"
token = "<my-token>"
# Store the URL of your InfluxDB instance
url="https://us-west-2-1.aws.cloud2.influxdata.com"
client = influxdb_client.InfluxDBClient(
url=url,
token=token,
org=org
)
# Write script
write_api = client.write_api(write_options=SYNCHRONOUS)
p = influxdb_client.Point("my_measurement").tag("location", "Prague").field("temperature", 25.3)
write_api.write(bucket=bucket, org=org, record=p)Based on this, we could update your code to include the new imports, add the new library to the requirements, add an environment variable inviting users to select between the two, and then - instead of using INFLUXDB_HOST, INFLUXDB_PORT, INFLUXDB_USERNAME, INFLUXDB_PASSWORD, and INFLUXDB_DATABASE to make the connection, we would do something like:
# In the includes section
import influxdb_client
from influxdb_client.client.write_api import SYNCHRONOUS
# In the portion of the code where you are reading environment variables
INFLUXDB_BUCKET = "your_bucket_name_here"
INFLUXDB_ORG = "your_org_here"
INFLUXDB_TOKEN = "your_token_here"
# Store the URL of your InfluxDB instance
INFLUXDB_URL ="http://your_url_here:8086"
# In the portion of the code where we we construct the database connector
try:
influxdbclient = influxdb_client.InfluxDBClient(
url=INFLUXDB_URL,
token=INFLUXDB_TOKEN,
org=INFLUXDB_ORG
)
# Write script
write_api = influxdbclient.write_api(write_options=SYNCHRONOUS)
except InfluxDBClientError as err:
logging.error("Unable to connect with influxdb database! Aborted")
raise InfluxDBClientError("InfluxDB connection failed:" + str(err))
def write_points_to_influxdb(points):
try:
write_api.write(bucket=INFLUXDB_BUCKET, org=INFLUXDB_ORG, record=points)
logging.info("Successfully updated influxdb database with new points")
except InfluxDBClientError as err:
logging.error("Unable to connect with influxdb database! " + str(err))
print("Influxdb connection failed! ", str(err))Happy to make a pull request with these changes - which would retain the default behavior of using the old influxdb library as the default behavior but give users the option to use the new one.
*** I think it's worth pointing out that I have made these changes from my end, and the system is working as expected.
Hello @signebedi, Thank you so much for your kind works and I really appreciate the information. I have a few quick points.
-
It looks like the new library is not compatible with the older versions of influxdb ( i.e. 1.8 etc, which is still widely used ), so we definitely need to keep backward compatibility. I see the library got archived, mostly because there is no possibility of update or change in the 1.x versions and they are not implementing the bucket system.
-
I personally cannot test this out because I am using influxdb 1.8 ( because it plays nice with grafana )
I think it's worth pointing out that I have made these changes from my end, and the system is working as expected.
That's good to know, I think it's always good to keep upgrade options open as long as it does not break any functionality. The major complicated part of the code is efficiently gathering the data and handling rate limits etc. The writing to database is the easy part there.
-
Does influxdb_client.InfluxDBClient raises the same InfluxDBClientError on error? remember that was imported from the other library. I don't see you importing that from the new library. Maybe take a look at this too. You did not encounter the connection error, hence it's working fine so far for you.
-
Users may get confused with both versions, so don't ask them to choose one. rather, use one for fallback of another. so in the env variable section, put the extra variables, and put a comment on the side like (# if you are using influxdb 2.x ) and put comment like (# if you are using influxdb 1.x ) for the variable I used, that way it's more clear. And they can fill up based on whatever they have. Now in the try section, try the connection with bucket method (influxdb 2.x ) first ( because it's more maintained), and if that raises connection error ( if the user kept those empty string ), try the connection with current variables ( the 1.x env variables ), and if that fails too then raise the error saying nothing worked. Let me know if that makes sense.
I will be very happy if you do a pull request given that i cannot test it myself. Thank you. Let me know if you have any questions.
I have updated the codebase to support both influxdb 1.x and 2.x in 529b031
Personally, I could not test the 2.x integration. I have ensured the backward compatibility of the new code with my influxdb 1.8.x
@signebedi will you mind testing out the current code and let me know if it works alright?
The docker container was updated subsequently, now it supports the updated ENV variables for 2.x integration.
Great, this looks about right. I am able to test the new version because I am running 2.x, but it might take me a few days to get to it.
Grafana works great with 2.x ... but, it uses flux instead of influxql... Personally, I enjoy using flux but can understand how its lack of backwards compatibility would be frustrating, to say the least.
@signebedi I have more information on this now. The flux support is going away in influxdb 3.x. influxdb 3.x will support influxQL natively so it will be backward compatible with 1.x. I am looking forward to it. In the meantime, influxdb 1.8 is actively maintained in the docker hub. I do not really see a reason to upgrade to 2.x now.
https://www.influxdata.com/blog/the-plan-for-influxdb-3-0-open-source/
Here is the new python library for influxdb 3.x, which I will integrate once the influxdb 3.x docker image is released.
Awesome, good catch. That's good to know. Bit frustrating that they went back-and-forth ... but I can understand the soul searching that these database applications are going through these days.
I will hold off on testing the changes for now and am happy to collaborate in the future on the new Python library to help make updates to this script.