SimBoi/deep-learning-hw3

CS236781: Deep Learning on Computational Accelerators

Homework Assignment 3

Faculty of Computer Science, Technion.

Introduction

In this assignment we'll learn to generate text with a deep multilayer RNN network based on GRU cells. Then we'll focus our attention on image generation using a variational autoencoder. We will then shift our focus to sentiment analysis: First by training a transformer-style encoder, and then by fine-tuning a pre-trained model from Hugging Face.

General Guidelines

• Please read the getting started page on the course website. It explains how to setup, run and submit the assignment.
• This assignment requires running on GPU-enabled hardware. Please read the course servers usage guide. It explains how to use and run your code on the course servers to benefit from training with GPUs.
• The text and code cells in these notebooks are intended to guide you through the assignment and help you verify your solutions. The notebooks do not need to be edited at all (unless you wish to play around). The only exception is to fill your name(s) in the above cell before submission. Please do not remove sections or change the order of any cells.
• All your code (and even answers to questions) should be written in the files within the python package corresponding the assignment number (hw1, hw2, etc). You can of course use any editor or IDE to work on these files.

Contents

• Part1: Sequence Models
  • Text generation with a char-level RNN
  • Obtaining the corpus
  • Data Preprocessing
  • Dataset Creation
  • Model Implementation
  • Generating text by sampling
  • Training
  • Generating a work of art
  • Questions
• Part 2: Variational Autoencoder:
  • Obtaining the dataset
  • The Variational Autoencoder
  • Model Implementation
  • Loss Implementation
  • Sampling
  • Training
  • Questions
• Part 3: Transformer Encoder
  • Reminder: scaled dot product attention
  • Sliding window attention
  • Multihead Sliding window attention
  • Sentiment analysis
  • Obtaining the dataset
  • Tokenizer
  • Transformer Encoder
  • Training
  • Questions
• Part 4: Fine-tuning a pretrained language model
  • Loading the dataset
  • Tokenizer
  • Loading pre-trained model
  • Fine-tuning
  • Questions