Getting past the difference in PVLive intra-day and PVLive updated

[dfulu]

We want to use some historic measurements to improve the accuracy of the model, particularly at short time horizons. However, using intra-day inputs to a model trained on updated PVLive decreases the accuracy of the model beyond ~1-hour time horizon (see #53). This is down to the differences between the initial estimate (intra-day) and the updated estimate. It's also important to note that after training PVNet we need to train the summation model so as to have probabilistic national prediction.

Assuming that we cannot get a significant backtest of GSP level PVLive intra-day estimates. Some solutions could be:

1. We could fine tune PVNet on the period we have intra-day PVLive for. However:
   • Its going to involve a bit or work to have a two step training process, along with two step batch creation
   • We only have intra-day values from 14th June 2022. So creating training and validation sets from this single year will involve some thinking. And we do need a validation set since currently we use early stopping to end training.
   • The NWP data we only have up to May-ish 2023 and the satellite only up to Mid-march 2023. So it reduces the time period we have to train/validate on during fine tuning.
   • We need to train the summation model on the outputs of PVNet. We may have to do a similar two-step training and fine-tuning for that model. It would also be a big data limitation for the summation model since we only get 1 sample per timestamp, unlike training PVNet where we get 317 samples per timestamp
2. We could train PVNet without PVLive inputs, but train the summation model with additional national intra-day inputs
   • We have a backtest of national intra-day PVLive inputs so less need to worry about differences
   • No need to fine tune
   • GSP level predictions may not be as good at short time horizons
   • The sum of our GSP predictions might differ significantly from our national prediction, especially at short time horizons
3. We try to use Passiv data instead of any PVLive inputs
   • Passiv data might give us all the effects of using PVLive intra-day inputs so that we don't need PVLive
   • However, there are still some issues with matching the IDs - which might take a while to solve
   • Passiv data has the chance to be a bit brittle if some of the systems stop reporting, so we need to be careful in training to use plenty of dropout etc

The easiest solution here would be 2. After solving issues with the Passiv data, we could combine 2 and 3

[dfulu]

@peterdudfield and @jacobbieker, any thoughts?

[peterdudfield]

Yea i agree, start with 2, and move on to 3 later.

Following on from your analysis earlier, might also be interesting to see if the accuracy per GSP improved by using PVlive intraday values

[jacobbieker]

Yeah, I also agree, 2 first, then 3

[peterdudfield]

Is there now a 4th option, which is to use PVlive intraday in training, now we have that from Sheffield Solar? I'd porbably go
2. then 4., what do you think?

[dfulu]

Yes that is an option now with the backtest data we have. I think I'd prioritise doing 4 and 2 together. 4 should be reasonably quick to do, its essentially just retraining a model after modifying the batches I already have saved out. 2 will be the same amount of work regardless of whether I do 4 or not

[dfulu]

I've been doing some sanity checks on the backtest of PVNet data - particularly looking at the national values.

Unfortunately the backtest we got from sheffield solar doesn't match the values in the database that we've been scraping from them in production. The comparison between the PVLive initial estimates we have in the database (i.e. the values we have at production time) and the backtest values are shown in the figure below. The backtest values are biased to be too high.

The figure below is trying to answer "are the backtest values closer to the initial estimates we have in production than the updated values?". Based on the graph below, I think the answer is yes, but probably not close enough to solve the issue in #53.

I think this likely means that we cannot do 4 without recreating the same issue as is #53. Similarly, 2 may not work for the same reasons.

[peterdudfield]

I've been doing some sanity checks on the backtest of PVNet data - particularly looking at the national values.

Unfortunately the backtest we got from sheffield solar doesn't match the values in the database that we've been scraping from them in production. The comparison between the PVLive initial estimates we have in the database (i.e. the values we have at production time) and the backtest values are shown in the figure below. The backtest values are biased to be too high.

The figure below is trying to answer "are the backtest values closer to the initial estimates we have in production than the updated values?". Based on the graph below, I think the answer is yes, but probably not close enough to solve the issue in #53.

I think this likely means that we cannot do 4 without recreating the same issue as is #53. Similarly, 2 may not work for the same reasons.

This may be because the backtest is run on the 1000 systems, but in relately perhaps only 700 sites are reporting live, and this could make a different.

hmm, perhaps the best way is to just use some PV Passive data, then we cut out the middle algorithm (and errors), this perhaps it simplier than using mulite GSP PVlive datasources

[dfulu]

Closing now as we do not plan to use PVLive inputs any longer and have already been experimenting with Passiv