bug: PVnet suddenly drops

Question

bug: PVnet suddenly drops

Closed this issue 8 months ago · 15 comments

peterdudfield commented 10 months ago

Detailed Description

PVnet seems to drop all froecasts. After running it again in 30 mins it seems fine

Context

see a blip in the 4 hour forecasts sometimes
NWP consumer on dev was only fixed about for the first 9.45 run

Answer 1 · 2023-08-25T11:58:21.000Z

peterdudfield commented 10 months ago

Answer 2 · 2023-08-30T11:16:09.000Z

Idea: could be missing data in satellite - that then gets filled in with zeros.

If we se again

check satellite data latest, check raw
check logs of pvnet2

Answer 3 · 2023-08-31T13:37:41.000Z

peterdudfield commented 10 months ago

Answer 4 · 2023-08-31T13:41:04.000Z

But I cant seem to find this is PVnet - might have to look at logs in blend service

Answer 5 · 2023-08-31T13:43:12.000Z

peterdudfield commented 10 months ago

Answer 6 · 2023-08-31T13:45:05.000Z

It was the 8.15 UTC run of PVnet 2

Answer 7 · 2023-08-31T13:46:42.000Z

Nothing unsual int he logs - https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logsV2:log-groups/log-group/$252Faws$252Fecs$252Fforecast_pvnet$252F/log-events/streaming$252Fforecast$252F2505f5d10f7b45509891cb2b804ea329$3Fstart$3D-43200000

Answer 8 · 2023-08-31T13:48:23.000Z

We see this in Braintree too, but not in other GSPs around it

Answer 9 · 2023-08-31T13:49:45.000Z

We do see this in GSPs 30 to 39 (inclusive) which is one batch of the PVnet run

Answer 10 · 2023-09-01T13:09:09.000Z

The batch was made and run at 8.17.51 to 8.17.57 UTC
No satellite data was uploaded then
NWP was uploaded but with no changes, at 8.17.36

Answer 11 · 2023-09-01T13:12:41.000Z

Another few today on dev

Answer 12 · 2023-09-01T13:17:18.000Z

We do see this in GSPs 30 to 39 (inclusive) which is one batch of the PVnet run

I think it would actually be batches of GSP IDs 1-10, 11-20, ... 31-40

Answer 13 · 2023-09-01T13:18:32.000Z

I worry its the NWP data upload that is interferrring in a small way. I could change the NWP data to run every 10,25,40,55 rather than 15,30,45,0, then its less likely to interferr

Answer 14 · 2023-09-01T13:40:47.000Z

We've seen more spikes in early gsp numbers that later ones, so this could correspond with NWP changing

Answer 15 · 2023-10-20T11:32:20.000Z

I think this was solved by the NWP download and the inversion fix. Closing for now