kdeyev/eyeonwater

Loading more granular historical data

Closed this issue · 46 comments

gulo101 commented

Hello,

It looks like the API pulls the data only few times a day, resulting in fairly incomplete data with low granularity. Would it be possible to load 15 min interval data historically and insert it to Home Assistant? Thanks

kdeyev commented

Yes, EyeOnWater seems to limit the number of HTTP requests per day.
You are right that I can get the history of the meter values, but HA sensor API does not allow me to modify the values of the sensor in the past.
If you have any idea how to implement it - I'm in :-)

bsp9493 commented

I'm sure someone noticed this before, but regardless of how often I poll with the integration, and regardless of my actual water usage, the reporting times always seem to be the same 7-8 am, 12-1am, 5-6pm, 7-8pm.

I know that I am using water outside of these time frames, and I also know that the integration is polling the service outside of these timeframes with no apparent errors.

Just seems strange that the reported timeframes of usage are always the same

kdeyev commented

Yes, It's the way EyeOnWater works: they read meters every 15 minutes and accumulate data somewhere in their backend, but they make the data available only several times a day.
So the meter data becomes not the immediate meter reads but meter history. Unfortunately, HomeAssistant does not really support sensors with historical data reads.
For example: at 6 PM they share all the meter reads for 12 PM-6 PM. Yes, data has a granularity of 15 minutes, but I cannot update the meter reads for 12 PM, because HASS does not support it.

bsp9493 commented
rfnovo commented

A possible solution would be for the sensor to collect hourly data for D-1 and then use the offset parameter of Utility Meter (https://www.home-assistant.io/integrations/utility_meter/) to adjust the data.
Historical data would be accurate, but we won't have same day data. but we already don't have real time data, so that would not be a huge drawback

kdeyev commented

The most feasible solution I see so far - modification of the sensor history in the HA DB. as it described here:
https://community.home-assistant.io/t/energy-counter-delayed-data/332108/12

I'm not sure regarding the robustness of the solution we could provide.

andrewchumchal commented

It does appear that homeassisant allow you to import historical data. If you take a look at this post here https://community.home-assistant.io/t/import-old-energy-readings-for-use-in-energy-dashboard/341406/9 he stats that it can be done with async_add_external_statistics and there are two plugins in the core that already use function as a example

I wonder if that is something that we use there at_a_glance API endpoint to be able to fill in the daily data

image

kdeyev commented

@andrewchumchal, It looks very promising. Will try to utilize this API as soon as I can.
Thank you for sharing.

kdeyev commented

Hi All,

Is there anyone interested to go through beta testing of the solution?
The fix was released here. Please feel free to enable pre-releases in HACS and test it.

kdeyev commented

Seems to work for me
Screenshot 2023-08-15 at 7 11 11 PM

bsp9493 commented
randomstring commented

I'm seeing the same error as @bsp9493

Logger: homeassistant.components.binary_sensor
Source: custom_components/eyeonwater/binary_sensor.py:88
Integration: Binary sensor (documentation, issues)
First occurred: 6:11:29 PM (1 occurrences)
Last logged: 6:11:29 PM
Error while setting up eyeonwater platform for binary_sensor

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 362, in _async_setup_platform
    await asyncio.shield(task)
  File "/config/custom_components/eyeonwater/binary_sensor.py", line 62, in async_setup_entry
    sensors.append(EyeOnWaterBinarySensor(meter, coordinator, description))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/eyeonwater/binary_sensor.py", line 88, in __init__
    name=f"Water Meter {self.meter.meter_info['meter_id']}",
                        ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'meter_id'
kdeyev commented

Yes, I'm facing the same issue. It was introduced recently and reported #27 .
Will fix it a bit later. For now, please reload the integration.

disforw commented

I am receiving the following:
Error occured during fetchig historinc data for sensor.water_meter_49281xxxxxxxxxxxxx8: Response is empty

kdeyev commented

@disforw Thank you for reporting!
I see the same and it started just after midnight. There is a period of time when the historical data is not collected/responded by EOW, so I'll need probably to suppress the error message.
BTW do you see the granular water usage data in your Energy Dashboard?

disforw commented

I do not, however, i installed the beta at 10:30pm Est, so if you’re right about midnight, I may have to wait for the next update.
OR, you’re stats were loaded on a previous update which is broken now. Try removing the integration and re-adding, see if your stats come back.

kdeyev commented

@disforw
While starting the integration should import 30 days of historical data, but it's visible only in the Energy Dashboard.
Screenshot 2023-08-15 at 7 11 11 PM
Could you please, check several past days if you see granular hourly data?
Also could you please check the "Developer Tools"->Statistics if there are any error messages for the water sensor?
Thank you!

disforw commented

I checked stats for the last 2 weeks, no granular data. No issues reported in developer tools.

bsp9493 commented
bsp9493 commented
kdeyev commented

I will need some assistance on your end.
It may happen that the request to EOW is not correct, for example, because of units.
Could you please run test.py for your account and see if you get any historical data from:

        data = await meter.get_historical_data(
            client=client, units="GAL", date=yesterday
        )
        for d in data:
            print(str(d["start"]), d["sum"])
bsp9493 commented
kdeyev commented

@bsp9493 It's definitely because of measurement units, we will need to find a proper way to request historical data in liters or m^3, I bet that EOW returns an empty response if the request is not supported.
Unfortunately, I can test only gallons since my account is in gallons.

The simplest way to run test.py is by downloading the latest source and running python test.py locally.
But it may be a bit of a challenge if you don't have a software dev background.

bsp9493 commented
bsp9493 commented
kdeyev commented

@bsp9493
Thank you for the effort!
I apologize, I forgot to mention that you need to use https://github.com/kdeyev/eyeonwater/tree/feature/import-statistics branch, since the changes were not been officially released yet.
Could you please take the source code from this branch and modify test.py again for your account, eow.ca, and metric system?
After you have the test.py adjusted, you don't really need any HAS stuff, test.py is self-contained.
Once you have the source code landed anywhere, you just need to run python test.py.
There are multiple ways to do it: probably the simplest is to use a command line terminal: cd source_code_location and python test.py.
Thank you!

bsp9493 commented
kdeyev commented

@bsp9493 Do you mind sharing temporary credentials for your EOW account for debugging purposes?
If you are ok with it, please contact me by kn.deev @ gmail

disforw commented

I am located in the US and using gal, but still no response. I also can not run the test bec I don’t have a proper dev environment. However, just before you started pushing this PR forward, I played around with your old code and successfully got a response from the API, so I moved the function to eow.py on Master. Let me play around a bit and get back to you.

the function in eow.py on master DOES respond with 24 hours of data.

kdeyev commented

@disforw Thank you for the answer.
Now I'm puzzled why the data is not imported by async_import_statistics(self.hass, metadata, statistics).
Usually home assistant prints error messages if anything goes wrong.
Or maybe metadata is somehow wrong on your end:

                metadata = StatisticMetaData(
                    has_mean=False,
                    has_sum=True,
                    name=name,
                    source="recorder",
                    statistic_id=statistic_id,
                    unit_of_measurement=meter.native_unit_of_measurement,
                )

I don't have any other ideas so far.

disforw commented

When I use the function that is currently in the master branch inside of eow.py called get_consumption, as it stands, I get back a promising result…
statistics = await meter.get_consumption(date=yesterday.strftime('%m/%d/%Y'), client=self.client) _LOGGER.debug(statistics)
Results to this are…


[{'start': datetime.datetime(2023, 8, 15, 23, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7266.26}, 
{'start': datetime.datetime(2023, 8, 15, 22, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7264.61},
 {'start': datetime.datetime(2023, 8, 15, 21, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7254.34},
 {'start': datetime.datetime(2023, 8, 15, 20, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7199.45},
 {'start': datetime.datetime(2023, 8, 15, 19, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7199.42}, 
{'start': datetime.datetime(2023, 8, 15, 18, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7196.51},
 {'start': datetime.datetime(2023, 8, 15, 17, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7196.22},
 {'start': datetime.datetime(2023, 8, 15, 16, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7162.2}, 
{'start': datetime.datetime(2023, 8, 15, 15, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7149.49}, 
{'start': datetime.datetime(2023, 8, 15, 14, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7146.36}, 
{'start': datetime.datetime(2023, 8, 15, 13, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7143.76},
 {'start': datetime.datetime(2023, 8, 15, 12, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7143.76}, 
{'start': datetime.datetime(2023, 8, 15, 11, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7143.74}, 
{'start': datetime.datetime(2023, 8, 15, 10, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7143.74}, 
{'start': datetime.datetime(2023, 8, 15, 9, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7138.16}, 
{'start': datetime.datetime(2023, 8, 15, 8, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7089.52}, 
{'start': datetime.datetime(2023, 8, 15, 7, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7080.47}, 
{'start': datetime.datetime(2023, 8, 15, 6, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7075.41}, 
{'start': datetime.datetime(2023, 8, 15, 5, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7075.41}, 
{'start': datetime.datetime(2023, 8, 15, 4, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7075.41}, 
{'start': datetime.datetime(2023, 8, 15, 3, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7074.26}, 
{'start': datetime.datetime(2023, 8, 15, 2, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7074.25}, 
{'start': datetime.datetime(2023, 8, 15, 1, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7074.23}, 
{'start': datetime.datetime(2023, 8, 15, 0, 0, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>), 'sum': 7071.02}]
kdeyev commented

That looks correct, does the 7074.23 gallons meter value correlate to the meter reads in EOW?
Could you please switch to the feature/import-statistics branch and verify it once again? (Just in case I messed something up in the branch).
Could you please also install the v1.3-beta8 to your HACS?
Do you see any suspicious logs related to the integration? Does the water meter sensor load ok and show the correct value? It also makes sense to look at Dev Tools -> Statistics for this sensor, it shows units misfit if there are.
The integration does not change the sensor history, but it updates the statistics associated with the sensor. It means that the result will be visible only in the standard Energy Dashboard. I just want no make sure that we are on the same page.

Here sensor history is not updated.
Screenshot 2023-08-16 at 5 41 00 PM
But the Energy Dashboard for the same day looks ok
Screenshot 2023-08-16 at 5 46 13 PM

disforw commented

Ok so it seems like you’re loading statistics in to a “sensor.water_meter_{meter_uuid}” when the actual entity_id being generated is “sensor.water_meter_{meter_id}”.
check your energy dashboard setup that you have the correct entity added there.

kdeyev commented

@disforw Yup, you are right, the ids were messed up. I fixed it and reconfigured the Energy Dashboard on my end.
It seems to work.
I've released v1.3-beta10. could you give it a try?

disforw commented

@kdeyev I am erroring out here, ive tried readding, but no good. Using beta 10

This error originated from a custom integration.

Logger: custom_components.eyeonwater
Source: components/recorder/statistics.py:2140
Integration: EyeOnWater (documentation)
First occurred: 10:38:42 PM (1 occurrences)
Last logged: 10:38:42 PM

Unexpected error fetching EyeOnWater data: Invalid statistic_id
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 283, in _async_refresh
self.data = await self._async_update_data()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 242, in _async_update_data
return await self.update_method()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/eyeonwater/init.py", line 68, in async_update_data
await eye_on_water_data.import_historical_data()
File "/config/custom_components/eyeonwater/coordinator.py", line 79, in import_historical_data
async_import_statistics(self.hass, metadata, statistics)
File "/usr/src/homeassistant/homeassistant/components/recorder/statistics.py", line 2140, in async_import_statistics
raise HomeAssistantError("Invalid statistic_id")
homeassistant.exceptions.HomeAssistantError: Invalid statistic_id

kdeyev commented

Fixed in v2.0.0

kdeyev commented

@disforw Ooops, I've just released v2.0

disforw commented

I think I got it.. My meter_id has letters in it. The entity-id is lower case and the statistic-id is upper case.

kdeyev commented

Ouch. I think it makes sense to switch to uuid, which will be unified.

disforw commented

So its hard to say, thatwould really depend on someone who has multiple meters on one account, which one is unique and which is account wide? To ME, it looked like meter-id was the per-meter unique id. I am basing this on the way the results were structured.

bsp9493 commented
kdeyev commented
bsp9493 commented
kdeyev commented

@bsp9493
I made a very naive fix and release it in v2.0.1-beta.1.
Also, do you mind opening another GitHub issue?