Pinned Repositories
lm-evaluation-harness
A framework for few-shot evaluation of language models.
fast_sequential_scan
A fast sequential scan on GPU
Flax-ResNets
CIFAR10 ResNets implemented in JAX+Flax
hawk-pytorch
PyTorch implementation of Hawk from "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models" (https://arxiv.org/abs/2402.19427). Compatible with torch.compile.
LeagueMatchScraper
Code to scrape League of Legends matches using the Riot Games API.
Little-GPT
GPT* - Training faster small transformers using ALiBi, Parallel Residual Connections and more!
RepVGG-CIFAR10
RepVGG models specifically for CIFAR10 and CIFAR 100. Based on RepVGG: Making VGG-style ConvNets Great Again (Ding et. al)
transformer_shmap
Tensor Parallelism with JAX + Shard Map
tritonformer
Trainable transformer with fwd+bwd ops in Triton, matching the performance of PyTorch + cuDNN/cuBLAS
ZeRO-transformer
Two implementations of ZeRO-1 optimizer sharding in JAX
fattorib's Repositories
fattorib/Little-GPT
GPT* - Training faster small transformers using ALiBi, Parallel Residual Connections and more!
fattorib/ZeRO-transformer
Two implementations of ZeRO-1 optimizer sharding in JAX
fattorib/Flax-ResNets
CIFAR10 ResNets implemented in JAX+Flax
fattorib/transformer_shmap
Tensor Parallelism with JAX + Shard Map
fattorib/LeagueMatchScraper
Code to scrape League of Legends matches using the Riot Games API.
fattorib/RepVGG-CIFAR10
RepVGG models specifically for CIFAR10 and CIFAR 100. Based on RepVGG: Making VGG-style ConvNets Great Again (Ding et. al)
fattorib/tritonformer
Trainable transformer with fwd+bwd ops in Triton, matching the performance of PyTorch + cuDNN/cuBLAS
fattorib/fusedswiglu
Fused SwiGLU Triton kernels
fattorib/StochasticDepthNets
PyTorch implementation of ResNet110 as described in Deep Networks with Stochastic Depth (Huang et al.)
fattorib/wtf-wikipedia-python
raw wikipedia XML to LM_Dataformat in under 4 hours
fattorib/Monte-Carlo-Fractal-Dimensionality
Efficient algorithm using random sampling to calculate the dimension of many basic fractals. Implemented algorithm in Python.
fattorib/picograd
picograd - Fully connected neural networks in Python
fattorib/fast_sequential_scan
A fast sequential scan on GPU
fattorib/GeometricDeepLearning
Introductory Geometric Deep Learning Presentation from September 2021
fattorib/hawk-pytorch
PyTorch implementation of Hawk from "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models" (https://arxiv.org/abs/2402.19427). Compatible with torch.compile.
fattorib/Python-Unigram
Unigram tokenization algorithm in Python
fattorib/CudaSoftmax
Softmax CUDA kernel :)
fattorib/fattorib.github.io
Website
fattorib/Fundamental-Domain
Code to generate a section of the fundamental domain for the action of the special linear group on the space of (integral) binary cubic forms. As it stands, the code is quite inefficient. In the future I hope to optimize it.
fattorib/InfoGAN-Jax
InfoGAN in Jax with small Gradio app
fattorib/jaxvae
Variational Autoencoder in JAX
fattorib/lm-evaluation-harness
Fork of lm-evaluation-harness for evaluating my custom models
fattorib/Python-BPE
I wrote Byte-Pair encoding but its 600x slower than 🤗
fattorib/ResNets-CIFAR10
PyTorch implementation of the CIFAR10 ResNets, based on Deep Residual Learning for Image Recognition (He et al.)