[slice_data] if date_to is in date format then is taking beginning of day

[BiancaMorandi]

When date_to is a date (e.g 2017-05-31), the slice_data function converts the date string into a datetime and therefore adds 00:00:00 to the date making it stop at 2017-05-31 00:00:00.

Expected behavior
Data should be taken up to the end of the day.

Note that:

• for brightdata we take 'less than' using df = df[df.index < to_date], e.g if date_to = '2017-06-01' then data is taken up to '2017-05-31 23:59:59'
• pandas is by default taking the end of the day if input is a date. e.g if date_to = '2017-06-01' then data is taken up to '2017-06-01 23:59:59'

Doing 'less than' the user does not need to remember how many days are in a particular month and so could just use '2017-06-01'. Whether we do 'less than' or not, the slice_data isn't consistent with that or even pandas so we need to change it.

[BiancaMorandi]

@stephenholleran Other functions using pd.to_datetime() and therefore having a similar issue are below:

• offset_timestamps function, if date_to is a date (e.g 2022-01-01) then the offset is applied up to 00:00 of the date (e.g 2022-01-01 00:00 ) and not up to the end of the day (e.g 2022-01-01 23:50)
• load_cleaning_file, if end date in cleaning file is set to a date e.g 2021-02-01 then data is cleaned up to previous timestamp of date, e.g 2021-01-31 23:50

Also bw.load.load._LoadBWPlatform.get_data(meas_loc_uuid, from_date=None, to_date='2021-05-01'), gets data up to '2021-05-01 00:00), but do not use pd.to_datetime() - this is what returned by platform.

bw.LoadBrightHub.get_data(measurement_station_uuid, date_from='2016-06-01', date_to='2016-07-02') is getting data up to 10 min before e.g '2016-07-01 23:50' as this is what returned by platform.

Do you want me to add an issue also for these functions?

[BiancaMorandi]

@stephenholleran we have discussed this issue internally and agreed to use the 'less than' method for date_to. I will make these updates as part of this issue.

[BiancaMorandi]

I have updated offset_timestamps functions to use 'less than' method for date_to. As part of this fix I have also updated the docstring, the examples, the tests and the bugs below.

1. In docstring it is stated that function can accept a datetime.time input but an error was raised when using this. This is fixed now.
2. In docstring it is stated that the output of the function is in same format as the input, this was not the case for datetime.datetime and datetime.date inputs. This is fixed now.

[stephenholleran]

Thanks @BiancaMorandi !