Confidence score for paligemma
Opened this issue · 7 comments
Hi @NielsRogge
I have finetuned my paligemma for custom data for image to JSON use case, but when I inference it some key values I got wrong like 3000 is extracted as 9000 so to get the data is correct or not, can i get Confidence score with paligemma?
Hi,
I'd recommend using a model with a higher image resolution (PaliGemma has models up to 896 resolution), provide more training examples, or train longer.
A confidence score can only be obtained for each token, as the model predicts one token at a time.
@NielsRogge thanks for your reply actually, i used cord dataset too, 800 images training dataset on g5.xlarge machine for 8 epochs because of early stopping it stopped ig , you tried with same dataset, but apart from increasing the training dataset quality, I just want a confidence score so that i can use it as reference, can you help me with that like confidence score and heatmap, btw once more thanks for your reply i really follow your pages, that really interest me
Ok, yeah I've not benchmarked it yet, it could be that a bigger model or models with higher image resolution perform better. I have similar notebooks on CORD for Donut, LLaVa and Idefics2, the latter has 8 billion parameters so for that you'd need an A100 for fine-tuning, but it might be that performance is also a lot better.
However it could also be that PaliGemma requires longer training
but in my case the training stopped due to early stopping ig should i increase the patience?, and are you planning to introduce confidence score for paligemma?
Yes you could increase patience, for confidence score see this answer: https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075?u=nielsr
Hi @NielsRogge
Could you also share us hardware requirements for your fine-tuning PaliGemma on CORD. I am getting memory issue using colab's T4 GPU. You have mentioned that for inference T4 GPU is sufficient. Also I would like to extract items like checkboxes efficiently in JSON, my main objective is to match the level of Google's DocumentAI. Could you suggest some datasets for fine tuning where my input files are more like application & survey forms and not receipts.
Hi @NielsRogge Could you also share us hardware requirements for your fine-tuning PaliGemma on CORD. I am getting memory issue using colab's T4 GPU. You have mentioned that for inference T4 GPU is sufficient. Also I would like to extract items like checkboxes efficiently in JSON, my main objective is to match the level of Google's DocumentAI. Could you suggest some datasets for fine tuning where my input files are more like application & survey forms and not receipts.
Hi @Sai-Monik I used g5.xlarge for finetuning it worked.