Bugazelle/export-csv-to-influx

time zone format for Date only?

Closed this issue · 2 comments

Hi, I have some CSV data I'm trying to import into influx with only a Time value of MM/dd/yy. I tried a bunch of formats but keep getting the error "Unexpected time with format"

tried all of these formats:

--time_format %MM/%dd/%yy
--time_format '%MM/%dd/%yy'
--time_format %MM/%dd/%yyyy
--time_format '%MM/%dd/%yyyy'

Hello @tswiftma ,

Try: --time_format %m/%d/%y

If you would like to know more python date format, check: https://strftime.org/

I have tried from my end with demo.csv:

timestamp,url,response_time
12/10/19,https://jmeter.apache.org/,1.434
12/10/19,https://jmeter.apache.org/,2.434
12/10/19,https://jmeter.apache.org/,1.200
12/10/19,https://jmeter.apache.org/,1.675
12/10/19,https://jmeter.apache.org/,2.265
12/10/19,https://sample-demo.org/,1.430
12/10/19,https://sample-show.org/,1.300
12/10/19,https://sample-7.org/,1.289
12/10/19,https://sample-8.org/,2.876

Command to run:

export_csv_to_influx \
	--csv demo.csv \
	--dbname demo \
	--measurement demo \
	--tag_columns url \
	--field_columns response_time \
	--user admin \
	--password admin \
	--force_insert_even_csv_no_update True \
	--server 127.0.0.1:8086 \
	--time_format %m/%d/%y

Data in influx:

> use demo
Using database demo
> show measurements
name: measurements
name
----
demo
> select * from demo
name: demo
time                response_time url
----                ------------- ---
1575936000000000000 2.265         https://jmeter.apache.org/
1575936000000000000 1.3           https://sample-show.org/
1575936000000000000 1.43          https://sample-demo.org/
1575936000000000000 2.876         https://sample-8.org/
1575936000000000000 1.289         https://sample-7.org/
>

There is a WARNING you may need to know:

Influx only could record one point for one timestamp.
See more info: influxdata/influxdb#2055

Because you use a Time value of MM/dd/yy, that means, some rows have the same time, and will cause some of rows will be "ignored" in influx.

The demo.csv has 9 rows, but only 5 rows saved in influx.
because:

  1. They share the same timestamp reason
  2. we use the --tag_columns url. If we don't pass any tag, only 1 row saved in influx (note: tag_columns is mandatory set)

if you would like to export all rows into influx, you should update --tag_columns to: --tag_columns url,response_time

So, for your data, to void influx ignore the data, you should add proper tag_columns based on your project.

Hope could be the help to you.

Merry Christmas to you ~~~

Bugazelle, this worked! Thanks for the detailed solution and have a great holiday :)

Tim