This project is meant for evaluating the summarization capabilities of three Large Language Models (T5-Small, T5-Large and GPT2).
CNN_Daily Mail Dataset [https://huggingface.co/datasets/cnn_dailymail]
ROUGE: Recall-Oriented Understudy for Gisting Evaluation (ROUGE) is a set of evaluation metrics designed for comparing summaries from Lin et al., 2004. See Wikipedia for more info. Here, we use the Hugging Face Evaluator wrapper to call into the rouge_score
package. This package provides 4 scores:
rouge1
: ROUGE computed over unigrams (single words or tokens)rouge2
: ROUGE computed over bigrams (pairs of consecutive words or tokens)rougeL
: ROUGE based on the longest common subsequence shared by the summaries being comparedrougeLsum
: likerougeL
, but at "summary level," i.e., ignoring sentence breaks (newlines)
- Single GPU available from Google Colab Free version
Deleted and cleaned the gpu memory after every evaluation so the GPU is not full.
del model
torch.cuda.empty_cache()
gc.collect() # Garbage Collector
![image](https://private-user-images.githubusercontent.com/43017632/257037492-a6bb5b04-6dcc-427e-a500-bb663702ab2c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTQ5ODEzNzAsIm5iZiI6MTcxNDk4MTA3MCwicGF0aCI6Ii80MzAxNzYzMi8yNTcwMzc0OTItYTZiYjViMDQtNmRjYy00MjdlLWE1MDAtYmI2NjM3MDJhYjJjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA1MDYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNTA2VDA3Mzc1MFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTI4OGEzMDhhYWMzYzA4ZGRlNmVlZjNjN2QwMTA5MjA4NjM5ZDM3YzBhMTk3YjcyYzM0YmNhMTJjYTc4ZjA5NzEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.ZGGBApeMntwhLyWf69DsOYsUAPWbZL8hw3gqRypimRM)
Model specific results and the summarization results are available in the attached notebook