singularity-energy/open-grid-emissions

Fix / clean up EIA-930 timestamp corrections

Opened this issue · 0 comments

The eia930.manual_930_adjust() function adjusts incorrect timestamps in the raw EIA-930 data:

def manual_930_adjust(raw: pd.DataFrame):

However, while poking around in this function as part of #300, it became clear that there are some potential issues with this function that need to be addressed:

  1. We currently are repeating adjustment functions and the code is quite messy and unorganized. It would be great if we could write a standardized functions to do these adjustments based on a specific start_datetime, end_datetime, and adjustment, and define a dictionary of all of these adjustment parameters, like in https://github.com/singularity-energy/scrapers/blob/master/scrapers/eia_genfuelmix.py#L87.
  2. When shifting the data to correct the timestamps, it matters whether you are shifting the data columns, or shifting the timestamp. For example, if you want to convert from a start of hour to end of hour convention, if you are shifting the data you need to shift by +1, but if you are shifting the timestamp you need to shift by -1 hour. It looks like generally we are shifting the data, but for the PJM interchange adjustment, it appears that we are shifting the datetime index instead and may be using the incorrect sign convention.
  3. The correction for TEPC data is out of date and needs to be updated (ie the issue has been corrected in the raw EIA-930 data).
  4. We should check that the timestamp corrections we are applying here are consistent with the timestamp corrections we are applying in the grid carbon API.