pretraining
There are 150 repositories under pretraining topic.
LlamaFamily/Llama-Chinese
Llama中文社区，Llama3在线体验和微调模型已开放，实时汇总最新Llama3学习资料，已将所有代码更新适配Llama3，构建最好的中文Llama大模型，完全开源可商用
microsoft/LMOps
General technology for enabling AI capabilities w/ LLMs and MLLMs
OFA-Sys/OFA
Official repository of OFA (ICML 2022). Paper: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
X-PLUG/mPLUG-Owl
mPLUG-Owl & mPLUG-Owl2: Modularized Multimodal Large Language Model
ChandlerBang/awesome-self-supervised-gnn
Papers about pretraining and self-supervised learning on Graph Neural Networks (GNN).
keyu-tian/SparK
[ICLR'23 Spotlight🔥] The first successful BERT/MAE-style pretraining on any convolutional network; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"
yuewang-cuhk/awesome-vision-language-pretraining-papers
Recent Advances in Vision and Language PreTrained Models (VL-PTMs)
YehLi/xmodaler
X-modaler is a versatile and high-performance codebase for cross-modal analytics(e.g., image captioning, video captioning, vision-language pre-training, visual question answering, visual commonsense reasoning, and cross-modal retrieval).
Alibaba-MIIL/ImageNet21K
Official Pytorch Implementation of: "ImageNet-21K Pretraining for the Masses"(NeurIPS, 2021) paper
qqlu/Entity
EntitySeg Toolbox: Towards Open-World and High-Quality Image Segmentation
dptech-corp/Uni-Mol
Official Repository for the Uni-Mol Series Methods
PKU-YuanGroup/LanguageBind
【ICLR 2024🔥】 Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
alibaba/Megatron-LLaMA
Best practice for training LLaMA models in Megatron-LM
PITI-Synthesis/PITI
PITI: Pretraining is All You Need for Image-to-Image Translation
PaddlePaddle/PaddleFleetX
飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。
michiyasunaga/LinkBERT
[ACL 2022] LinkBERT: A Knowledgeable Language Model 😎 Pretrained with Document Links
microsoft/AzureML-BERT
End-to-End recipes for pre-training and fine-tuning BERT using Azure Machine Learning Service
amazon-science/bigdetection
BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-training
j-min/VL-T5
PyTorch code for "Unifying Vision-and-Language Tasks via Text Generation" (ICML 2021)
microsoft/UniVL
An official implementation for " UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation"
X-PLUG/ChatPLUG
A Chinese Open-Domain Dialogue System
archersama/awesome-recommend-system-pretraining-papers
Paper List for Recommend-system PreTrained Models
OpenGVLab/PonderV2
PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm
michiyasunaga/dragon
[NeurIPS 2022] DRAGON 🐲: Deep Bidirectional Language-Knowledge Graph Pretraining
showlab/UniVTG
[ICCV2023] UniVTG: Towards Unified Video-Language Temporal Grounding
phellonchen/awesome-Vision-and-Language-Pre-training
Recent Advances in Vision and Language Pre-training (VLP)
Coobiw/MiniGPT4Qwen
Personal Project: MPP-Qwen14B(Multimodal Pipeline Parallel-Qwen14B). Don't let the poverty limit your imagination! Train your own 14B LLaVA-like MLLM on RTX3090/4090 24GB.
kingTLE/literary-alpaca2
从词表到微调这就是你所需的一切
uta-smile/TCL
code for TCL: Vision-Language Pre-Training with Triple Contrastive Learning, CVPR 2022
akanyaani/gpt-2-tensorflow2.0
OpenAI GPT2 pre-training and sequence prediction implementation in Tensorflow 2.0
guolinke/TUPE
Transformer with Untied Positional Encoding (TUPE). Code of paper "Rethinking Positional Encoding in Language Pre-training". Improve existing models like BERT.
westlake-repl/SaProt
Saprot: Protein Language Model with Structural Alphabet
linjieli222/HERO
Research code for EMNLP 2020 paper "HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training"
showlab/EgoVLP
[NeurIPS2022] Egocentric Video-Language Pretraining
ZigeW/data_management_LLM
Collection of training data management explorations for large language models
fajieyuan/SIGIR2020_peterrec
Universal User Representation Pre-training for Cross-domain Recommendation and User Profiling