task: Ichigo LLM v0.5 Training
hahuyhoang411 opened this issue · 6 comments
hahuyhoang411 commented
Goal
Experiment on Ichigo model on multilingual.
Tasklist
- #125
- Experiment with hyper-parameters
- Draft research report
- Check for the degradation
- Check the model can do multi-lingual conversation
hahuyhoang411 commented
Lora for pretraining ref: https://unsloth.ai/blog/contpretraining
tuanlda78202 commented
Paired with @bachvudinh to understand the fully trained original Ichigo pipeline.
dan-menlo commented
@hahuyhoang411 - Can we combine this into #116