Temporal Fusion Transformers - overview

About The Project

The repository contains code to understand and test the state-of-the-art architecture for Interpretable Multi-horizon Time Series Forecasting, proposed by Google in 2020. This mini-project was created for a recruiting assignment.

Full article name: Temporal Fusion Transformers for interpretable multi-horizon time series forecasting

Authors: Bryan Lim, Sercan Arik, Nicolas Loeff and Tomas Pfister

Code Organisation

data consists of data files: both created with the help of the Google tutorial and generated manually (economicy/ directory)
google_repo directory contains files directly copied from the official Google Research repository
google_repo_old consists of similar, but a bit different scripts as in google_repo - created in first attempt of solving the problem
manually_download_data concerns data dedicated specifically for the project and scripts formatting them, together with external data detailing the dates
saved_models includes automatically generated files after running script_train_fixed_params program

Out of four available datasets prepared by Google for testing their solution, three were successfully downloaded to the hard drive (except for favorita), while only volaitte is available on GitHub due to system's limitations (no more than 100 MB per file). Others have to be downloaded manually with the help of a tutorial available on the official repository.

Useful links

Recruitment task data

In order to examine TFT, three important factors were analyzed [together with their later abbreviated forms]:

a) The Standard and Poor's 500 - SP500

b) Gold Price in USD - Gold

c) EUR to USD Exchange Rate - EURUSD

All indexes were downloaded from the MarketWatch website. The data thus obtained contains some NA values, but fortunately mostly on weekends. Another limitation was the inability to download a larger range of data than one year - in order to sort it out, ten (for ten years) .csv files were manually downloaded for each of the three indexes, which were then processed and merged.

Each day and index consists of four numbers:

i) open - the value at the beginning of the day

ii) high - maximum value of the considered day

iii) low - minimum value of the considered day

iv) close - end-of-day value

Preprocessing data

After downloading data, a few actions have been made in manually_downloaded_data/format_manually_downloaded.ipynb script. The code consists operations such as:

connecting 30 manually downloaded files
having all the data in one Pandas DataFrame, containing all the dates since April 2nd, 2012 to April 1st, 2022
generating summaries and visualizations, including correlation matrix and NA value information
merging dataset containing all the available daily data from last 10 years with detailed dates' data frame
some extra operations at the end of the code, with which I tried to solve the potential problem of bad input data format for the algorithm (removing NA values, adding artificial category, backward fill; in different configurations)

Problems with Google repository

Original scripts cloned contains some bugs, mostly caused by changes in new versions of TensorFlow, for which the code has not yet been adapted. Therefore, in many places the code is modified to work as of April 2022.

Version 0.1 - Sunday, April 3rd

Among others, due to chenges in TensorFlow, in all the scripts importing package line:

import tensorflow as tf

... was changed to:

import tensorflow.compat.v1 as tf

... with addition of:

tf.compat.v1.experimental.output_all_intermediates(True)

in a few scripts. Also, some packages had to be installed to old versions - eg. scikit-learn to 0.24.2. In the end it was again updated to the latest version, but this did not affect the result.

Version 0.2 - Thursday, April 7th

New idea came to my mind - maybe old versions of TenserFlow are not compatible with part of the code? That's why I changed my idea.

Second gooogle_repo was copied, once again from original source. The old one was renamed to google_repo_old and in this variant I decided to import two TenserFlows package:

import tensorflow as tf
import tensorflow.compat.v1 as tf1

In most of TenserFlows references I use the first one. tf was changed to tf1 only in cases of error attribute errors, which were repaired by referencing to the old version.

However - the final effect ended up the same :(

Traceback (most recent call last):
  File "C:\Users\koste\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\koste\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\koste\Studia\10\JOB\7bulls\tft-overview\google_repo\script_train_fixed_params.py", line 234, in <module>
    main(
  File "C:\Users\koste\Studia\10\JOB\7bulls\tft-overview\google_repo\script_train_fixed_params.py", line 153, in main
    val_loss = model.evaluate(valid)
  File "C:\Users\koste\Studia\10\JOB\7bulls\tft-overview\google_repo\libs\tft_model.py", line 1191, in evaluate
    raw_data = self._batch_data(data)
  File "C:\Users\koste\Studia\10\JOB\7bulls\tft-overview\google_repo\libs\tft_model.py", line 765, in _batch_data
    data_map[k] = np.concatenate(data_map[k], axis=0)
  File "<__array_function__ internals>", line 180, in concatenate
ValueError: zero-dimensional arrays cannot be concatenated

The above discussion suggests that the problem is in the split_data function from the Economy DataFormatter, whereas I don't think that is the problem.

Other users' codes

Despite forking the Google repository by several thousand users, I only found any changes to the /tft directory itself in a few cases. None of these, however, contained an implementation of any other than four of the proposed datasets...

However, it turned out that the code itself was implemented in a different way! The solution was at least prepared by user KalleBylin, as Master of Data Science ML Project. His solution which is available in one .ipynb file and seems to work, can be found at this link. I included it also in this repositorium, named KatteBylinSolution.ipynb in the main directory.

End of the day - at least on my computer - one epoch did not count in 12 hours :(

Kostee/tft-overview