Overall Config Setup

[Oufattole]

Support for Modular Configuration with Early and Late Fusion Options

Problem

Current model configurations lack the flexibility to easily incorporate and experiment with early and late fusion techniques, which are crucial for enhancing model performance by integrating information at different stages of the processing pipeline.

Proposed Solution

Develop a modular configuration that supports user defined models and losses:

• Data Processing: Similar initial data processing step.
• Input Encoder: Encode data as in the standard pathway.
• Model: A more flexible stage where the model can implement any form of fusion (early or late) and handle data labeling internally according to specific experimental needs. The inputs to this stage are a sequence model architecture, for example an LSTM or a transformer decoder, and it takes the output of the input encoder as input. if there is a specific pretraining task (such as performing early fusion and shuffling windows for OCP, that is performed at this stage). Pretraining and finetuning models will be seperate files, and weight loading should be supported between them