For this project I am using 'bert-base-cased'. It's pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in the paper and first released in this repository. The model is case-sensitive: it makes a difference between english and English.
I used opendelta library for parameter efficient finetuning
> Adapter
trainable params: 903744 - all params: 109216323 - trainable: 0.827481% > LoRA
trainable params: 294,912 - all params: 108,607,491 - trainable: 0.2715392808402139%Well LoRA and QLoRA are very good for LLM I thought of using it in BERT to get a good understanding of how LoRA, Adapter work and I used LoRA for two reasons. The first was curiosity to know how the model would perform with a trainable percentage as low as 0.27%. The second reason was the lack of electricity and internet, so I needed faster training. Despite this, the resources did not allow me to see results. Final training.
In this project I using scrapped lyrics from 6 genres and I selected Rap, Rock and Hip Hop
I did not have the resources, such as the Internet, electricity, device, etc., to train the model well and choose the appropriate learning rate, so there were no results.
To contribute to the project, please contribute directly. I am happy to do so, and if you have any comments, advice, job opportunities, or want me to contribute to a project, please contact me V3xlrm1nOwo1@gmail.com