This is still a work in progress, but should be ready some time in November. For now, if you’re interested in these topics you might want to check out the previous course I ran on AI ART: AIAIART.
If you want to be notified when the course goes ‘live’ you can sign up for an email at http://thegenerativelandscape.com/
If you want to support this effort, I now have a patreon: https://www.patreon.com/johnowhitaker
The material will show in github pages at https://johnowhitaker.github.io/tglcourse/ (and the http://thegenerativelandscape.com will redirect there once we launch). Hooray for the magic of nbdev.
The idea is to have a core curriculum building up to an understanding of key generative modelling techniques, split into three rough sections. The first 5 lessons will cover basics of building NNs and crafting loss functions. Lessons 6-10 will introduce generative modelling, GANs and working with sequences using transformers. Lessons 11-15 will be a deep dive into diffusion models, and lesson 16 will wrap up and give sugestions for new directions to explore.
Alongside this will be a number of ‘bonus’ notebooks that don’t need to be completed but which augment the core content. Managing datasets, experiment tracking, sharing demos and so on. Some will augment specific lessons, some will add functionality to the library and some will just be standalone topics I think are cool. The latter category is likely to continue to grow even after the course launches :)
There will be three suggested projects (after lessons 5, 11 and 14) to mark milestones in the course, and the final lesson will also encourage you to do a larger project at the end. Still working on details for how we’ll share and encourage these student projects.
This table has a rough status on the main lessons. It looks more drastic than it is - for many of these I have content lurking in other places so even those marked ‘Not Started’ should come together a lot faster than if this were being done completely from sratch!
| Lesson | Description | TODOs | Status |
|---|---|---|---|
| Lesson 1: PyTorch Basics | Intro to PT, tensor manipulation, images as tensors, LeastAverageImage exercise | Video, tidy | Rough Draft Done |
| Lesson 2: Optimization | Intro to GD, optimization examples exercise | Video, Optimize with gen+loss without explaining | Rough Draft Done |
| Lesson 3: Building NNs | nn.Module, building blocks, CNNs | Video, project, export functions? optimization/training tricks? | WIP |
| Lesson 4: Learning Representations | What do networks learn, style transfer | Video, tidy, … | WIP |
| Lesson 5: CLIP | Contrastive loss, CLIP, as a loss function | everything (pull from AIAIART) | Not Started |
| Lesson 6: Generative Modelling | Intro to generative models, AEs and VAEs, issues | everything (pull from AIAIART) | WIP |
| Lesson 7: GANs 1 | Intro to GANs, DC GAN, Conditioning | everything | Not Started |
| Lesson 8: GANs 2 | GAN training tricks, NOGAN, using modern GANs, VQGAN | everything | Not Started |
| Lesson 9: Sequence Modelling Intro | idea, language modelling concept, transformer demo | everything | Not Started |
| Lesson 10: Transformers | Intro to transformes, attention, from scratch or dissect existing ones? ViT? | everything, check Niels’ repo and other resources | Not Started |
| Lesson 11: Everythign is a sequence | Show whistlegen, protein, VQGAN, parti… | everything | Not Started |
| Lesson 12: DM 1 | Intro to diffusion models, toy example, comparison to DDPM | Video, tidy, second half, logs of my version training, better unet replacement | Rough Draft Done |
| Lesson 13: DM2 | Conditioning, CFG, guiding, sampling, better training | everything | WIP |
| Lesson 14: DM3 | SD deep dive | link to video run-through or record smaller videos | Rough Draft Done |
| Lesson 15: DM4 | Other modalities - ideally demo class-conditioned audio generation. Might not be done by the time the course launches but would be nice to have. | wavenet style diffusion for birdcalls, look at video stuff | Not Started |
| Lesson 16: Going Further | Finding your niche, exploring less common areas | TODO | Not Started |
General TODOs - Create a page for project explanations + figure out submission info. - Figure out what the discussions page (pages) looks like - Ethics page but also callouts in each lesson - Pick a cloud for cloud stuff (sponsor?) - Figure out plan for datasets - Figure out losses+generators notebook/library bit and include in the plan somewhere DONE - Figure out where smaller ‘bonus’ things like paper readings or topical deep-dives go. Do we add a blog? - Make the website pretty - Generate a bunch of images - Get Dave to make video assets + logo - Streamline skipping tests / avoiding anything long-running during CI - Set up org on HF for sharing models - Set up W&B projects for sharing training runs/reports - Notebook to document the library
- Datasets and Dataloaders Intro (also defines data util funcs for the rest of the course)
- Experiment Tracking (eg W&B)
- Pretrained networks and transfer learning
- Ethics in generative modelling
- Fine-tuning existing models
- Working with GPUs? Multi-GPU or TPU training?
- Sharing demos w/ Gradio
- Managing cloud machines
- Dataloaders deep dive, streaming data
- Navigating other codebases
- Version control and CI (+ NBDev)
- “Text inversion” training (https://arxiv.org/pdf/2208.01618.pdf)
- “prompt to prompt editing” https://arxiv.org/pdf/2208.01626.pdf
- Lots of paper readings / sumaries
- Neural cellular automata
- Evolutionary algorithms
- Create a custom generation pipeline by combining different generators and loss functions
- Train a GAN, explore hyperparameters
- Fine-tune a diffusion model on a custom dataset and share it with the community
- Create and share a final project, including a report and demo
Guest discussions: as many as we can :)
Paper explainers: as many as we can :)
Page stats: Total Hits:
