[Data Leakage] Growth Ratio/Factor transformation before train-test split leaks data.

Question

[Data Leakage] Growth Ratio/Factor transformation before train-test split leaks data.

Closed this issue 4 years ago · 0 comments

Pretty subtle form of leakage in the Growth ratio and Growth Factor sections :

I have the confirmed cases time-series for n days:
$C_n$

Then I calculate the Growth Ratio for nth day:
${Gr_n}= \frac{C_n}{C_{n-1}}$

and the plan is to model growth ratio and predict growth ratio on day n+1
$\hat{Gr_{n 1}} = \frac{\hat{C_{n 1}}}{C_{n}}$

which can then be simply transformed into the predicted cases:
$\hat{C_{n 1}}= \hat{Gr_{n 1}}\times{C_n}$

The problem is when the Gr is calculated on the entire dataset before train-test split.

Ex, suppose you use cal. Gr and use first 10 days for train and 11... days for validation. Gr on 10th day:
Gr on 10th day:
$Gr_{10} = \frac{C_{10}}{C_{9}}$

and for validation set, the Gr on day 11th:
$Gr_{11} = \frac{C_{11}}{C_{10}}$

There is overlap, and now if you use lag features $Gr_{t-1},Gr_{t-2}\dots{}$

The leakage is amplified.

[This is not a problem if time-series is not for a variable that is a ratio(encodes backwards data) then calculate lag and forward multi-step recursive forecast is fine]

Solution:
Split original series into train-test and then calculate the growth ratio and create lags. This loses 'number of lag' samples from start of validation set but should be leakage free.

Fixed for growth ratio, skipped for growth factor.