Measuring GenAI Models

Repository with experiments done for a course "Generative Neural Networks for the Sciences" at the University of Heidelberg.

Authors: Agata Kaczmarek and Mateusz Stączek

Note:

Some of the notebooks were executed in the Google Collab environment, so the paths to the files are set up accordingly. If you want to run the notebooks locally, you will need to adjust the paths to the files.

Data - generated images for download

All zips with images can be downloaded from the Onedrive link below (expires on Saturday, Jun 22, 2024):

https://wutwaw-my.sharepoint.com/:f:/g/personal/01151437_pw_edu_pl/Egsg7WUcVN5Enbj7RYM2bnkBrIgUKZM8_fPIdl5-slcchw?e=PLYxJG

Contents (names of files start with Step number from the report)

Sample usage of various models found in preparation for the project.
- 0_findings.ipynb
DALL-E 3 and SDXL Turbo - generating images
- 2_generate_images.ipynb
Computing predictions of CNN and YOLO models on images from DALL-E 3 and SDXL Turbo, and for StyleGAN 2 in a separate notebook.
- 3_5_cnn_yolo_pipeline.ipynb
- 11_cnn_yolo_pipeline-GAN.ipynb
Confidence of hypothesis testing with YOLO.
- 6_analyzing_yolo_detection.ipynb
- 11_analyzing_yolo_detection_GAN.ipynb
Confidence of hypothesis testing with CNN.
- 4_analyzing_cnn_classification_results.ipynb
- 11_analyzing_cnn_classification_GAN.ipynb
Compute captions with BLIP image-to-text model.
- 7_image-to-text-our-images.ipynb
Compute cosine similarity between captions and prompts.
- 8_prompt-caption_similarity.ipynb
Compute FID metrics for StyleGAN 2 images.
- 12_fid_GAN_metric.ipynb

Other files, prompts, csvs

Folder hypothesis_testing_csv_tables contains CSVs with results of hypothesis testing for CNN and YOLO models for DALL-E 3, SDXL Turbo and StyleGAN 2 images.
Folder datasets_generated_ready_models contains, in respective subfolders, all results such as classification results, captions, and object detection results.
Original prompts are in the original prompts folder.

Sample images