[Question] Are numerical and categorical features used for fine-tuning the BERT(or LLMs)?
Closed this issue · 3 comments
anirbandey303 commented
Hi Developers,
I have a naive question, could you please help me understand:
- During the model training, are the weights and biases of the BERT model(or any other supported model) changing? (IF yes,) is it taking the numerical and categorical features into account or is it being used as a feature (along with the vector-embeddings from the LLM) by the MLP that does the classification?
akashsaravanan-georgian commented
Hi @anirbandey303,
Yes, the pre-trained language model (such as BERT) weights do change.
The numerical/categorical features are not used by the language (BERT) model. They are sent as features to the MLP alongside the output of the language model. The architecture diagram here might help!
anirbandey303 commented
Perfect, that clears out my confusion. Thanks a lot for the prompt response. 👍
akashsaravanan-georgian commented
Happy to help :)