georgian-io/Multimodal-Toolkit

[Question] Are numerical and categorical features used for fine-tuning the BERT(or LLMs)?

Closed this issue · 3 comments

anirbandey303 commented

Hi Developers,

I have a naive question, could you please help me understand:

  • During the model training, are the weights and biases of the BERT model(or any other supported model) changing? (IF yes,) is it taking the numerical and categorical features into account or is it being used as a feature (along with the vector-embeddings from the LLM) by the MLP that does the classification?
akashsaravanan-georgian commented

Hi @anirbandey303,

Yes, the pre-trained language model (such as BERT) weights do change.

The numerical/categorical features are not used by the language (BERT) model. They are sent as features to the MLP alongside the output of the language model. The architecture diagram here might help!

  • 🚀1
anirbandey303 commented

Perfect, that clears out my confusion. Thanks a lot for the prompt response. 👍

akashsaravanan-georgian commented

Happy to help :)

  • ❤️1