abhinand5/tamil-llama

Regarding Training of Model

Opened this issue · 1 comments

Cyberblackstone commented

We are currently working on a project where we want to finetune the current Tamil lama model to create an application for a domain specific use, which requires specialised vocabulary and terminology in the Tamil Language.

My question relates to the training of the Tamil LLama Model. Do I have have to pretrain the already fine tuned model, or do I have to start with the Meta LLama2 model and then use your tamil data set? What are the steps that are necessary to begin training/fine-tuning the Tamil Llama model with a custom data set?

Currently we are building a corpus of our domain specific tamil data, and this is what we ultimately want to fine tune. Currently the Tamil Llama model works really well, however we have been facing difficulties in getting the model fine tuned, any step provided in this direction would be great!!

Thanks in advance.

akash-balakrishnan commented

I believe you don't want to pre-train the model. Since you are building your own custom dataset, if you're data samples are more than 1000 samples, my recommendations would be:

  1. Use the Tamil-llama base model.
  2. Apply techniques like PEFT or LoRA on your custom dataset based on your use case.