My Bachelor Graduation Design

On the lonely road of coding, long and winding,
Poverty's shackles often leave hearts pining.
But I believe, despite all the pain and sorrow,
My thesis shall bring forth a brighter tomorrow.

PyTorch_For_PoorGuys, a name not renowned,
Yet with dreams in my heart, aspirations unbound.
Harnessing CPU might to transcend VRAM's plight,
May my design illuminate, dispelling the night.

In the ocean of tech, I journey with zest,
Exploring, challenging limits, never at rest.
May my thesis, like the North Star, guide the way,
Breaking poverty's chains, bringing forth a new day.


Poem written by OpenAI's ChatGPT 3.5

PyTorch_For_PoorGuys: A Deep Learning Framework for Training Large-Language-Model on Cheap Workstation with Smaller VRAM required.

NVIDIA GeForce 1080Ti/2080Ti/3090/4090 are cheaper for Universitys or personal researchers.

LLM was famous and Omoshiroii since OpenAI GPT-3 proposed.

However, caused by its smaller Video-RAM, it is hard to train a LLM on orginal PyTorch.

I want to let GPU borrow some Host-RAM to buffer LLM data when its VRAM has been exhausted.

So I do a fork with original PyTorch and modified its code to enable DMA method.

You can download the prebuilt wheel installer file of the hacked version of PyTorch and compatible Torchvision build from here, or build it yourself from source code!

Features

Allocated Memory Buffer on VRAM first for Lower Latency, once the VRAM exhausted, it automatic allocated from Host RAM
Create a Memory Pointer which can be both accessed from CPU or GPU side (Need hacking code yourself if you want completely using DMA)
With an easy version of Website working around with a Toy ChatGPT pretrained by this framework

Future

Hacking the GPU Driver to DIY Memory-Management-Unit, enable more flexable way to play with GPU
Let GPU controlling Hardware (Like Servo Motors) using Host Physic-Address via DMA directly
Use FPGA to design a Physics Address Listener via PCI-E and DMA to interaction with GPU
It will be interesting when Sensor signal can be passed to the Deep-Learning-Model in parallel directly, and the Model output signal can controlling device in parallel directly, just let CPU to sleep!

Hardware Requirement

NVIDIA Gefore 1080Ti/2080Ti/3090/4090 Graphic Card
Compatible motherboard and Power Supply
Intel Xeon E5-2620 V4 or Higher CPU
128GB DDR4 RAM or Higher
1TB SSD or Higher

Software Requirement

Debian Style of Linux, Ubuntu 20.04 LTS is suggested
PyTorch 2.0 Compatible GPU Driver
PyTorch 2.0 Compatible CUDA Developer Toolkit. See Build sections for details in Backend modules README.md file
A GUI environment with browser to run frontend HTML code (Can be on your laptop or another computer)

File Directory Structure

Backend
- PyTorch_For_PoorGuys: The Core Module I have been proposed.
- DeepSpeedExamples: The LLM training & infering code that I forked and modified from Microsoft's Repo
Frontend
- playground.html: The HTML & Javascript Demo site that connect to Backend.
- dummyBackend.py: The Python & Flask code that mook Backend response to debugging Frontend

Sync the Repo

git clone --recursive https://github.com/UEFI-code/BachelorGraduationDesign

Or if you have already cloned the repo, you can sync the submodules by:

git submodule update --init --recursive

If the sync of submodules has error, then you can try to reset the submodules by:

git submodule foreach git reset --hard

Notice

This code will not Allocate Pageable Host Memory, which means it occupy Physic RAM at runtime
Never pass a Memory Pointer which need CPU do Exception-Handling to GPU, or you will get crash