pre-training
There are 161 repositories under pre-training topic.
RUCAIBox/LLMSurvey
The official GitHub page for the survey paper "A Survey of Large Language Models".
modelscope/ms-swift
Use PEFT or Full-parameter to finetune 400+ LLMs (Qwen2.5, Llama3.2, GLM4, Internlm2.5, Yi1.5, Mistral, Baichuan2, DeepSeek, ...) or 100+ MLLMs (Qwen2-VL, Qwen2-Audio, Llama3.2-Vision, Llava, InternVL2.5, MiniCPM-V-2.6, GLM4v, Xcomposer2.5, Yi-VL, DeepSeek-VL2, Phi3.5-Vision, GOT-OCR2, ...).
modelscope/data-juicer
Making data higher-quality, juicier, and more digestible for foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷为大模型提供更高质量、更丰富、更易”消化“的数据！
dbiir/UER-py
Open Source Pre-training Model Framework in PyTorch & Pre-trained Model Zoo
ChandlerBang/awesome-self-supervised-gnn
Papers about pretraining and self-supervised learning on Graph Neural Networks (GNN).
EgoAlpha/prompt-in-context-learning
Awesome resources for in-context learning and prompt engineering: Mastery of the LLMs such as ChatGPT, GPT-3, and FlanT5, with up-to-date and cutting-edge updates.
LirongWu/awesome-graph-self-supervised-learning
Code for TKDE paper "Self-supervised learning on graphs: Contrastive, generative, or predictive"
zjunlp/KnowLM
An Open-sourced Knowledgable Large Language Model Framework.
yzhuoning/Awesome-CLIP
Awesome list for research on CLIP (Contrastive Language-Image Pre-Training).
Tencent/TencentPretrain
Tencent Pre-training framework in PyTorch & Pre-trained Model Zoo
microsoft/Oscar
Oscar and VinVL
SalesforceAIResearch/uni2ts
Unified Training of Universal Time Series Forecasting Transformers
brightmart/bert_language_understanding
Pre-training of Deep Bidirectional Transformers for Language Understanding: pre-train TextCNN
qingsongedu/Awesome-TimeSeries-SpatioTemporal-LM-LLM
A professional list on Large (Language) Models and Foundation Models (LLM, LM, FM) for Time Series, Spatiotemporal, and Event Data.
ChenRocks/UNITER
Research code for ECCV 2020 paper "UNITER: UNiversal Image-TExt Representation Learning"
jackroos/VL-BERT
Code for ICLR 2020 paper "VL-BERT: Pre-training of Generic Visual-Linguistic Representations".
nancheng58/Awesome-LLM4RS-Papers
Large Language Model-enhanced Recommender System Papers
princeton-nlp/LLM-Shearing
[ICLR 2024] Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
Shen-Lab/GraphCL
[NeurIPS 2020] "Graph Contrastive Learning with Augmentations" by Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, Yang Shen
acbull/GPT-GNN
Code for KDD'20 "Generative Pre-Training of Graph Neural Networks"
microsoft/XPretrain
Multi-modality pre-training
GAIR-NLP/MathPile
[NeurlPS D&B 2024] Generative AI for Math: MathPile
linwhitehat/ET-BERT
The repository of ET-BERT, a network traffic classification model on encrypted traffic. The work has been accepted as The Web Conference (WWW) 2022 accepted paper.
google-research-datasets/conceptual-12m
Conceptual 12M is a dataset containing (image-URL, caption) pairs collected for vision-and-language pre-training.
GestaltCogTeam/STEP
Code for our SIGKDD'22 paper Pre-training-Enhanced Spatial-Temporal Graph Neural Network For Multivariate Time Series Forecasting.
THUDM/GCC
GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training @ KDD 2020
westlake-repl/Recommendation-Systems-without-Explicit-ID-Features-A-Literature-Review
Paper List of Pre-trained Foundation Recommender Models
sayakpaul/probing-vits
Probing the representations of Vision Transformers.
Lupin1998/Awesome-MIM
[Survey] Masked Modeling for Self-supervised Representation Learning on Vision and Beyond (https://arxiv.org/abs/2401.00897)
ZigeW/data_management_LLM
Collection of training data management explorations for large language models
ViTAE-Transformer/SAMRS
The official repo for [NeurIPS'23] "SAMRS: Scaling-up Remote Sensing Segmentation Dataset with Segment Anything Model"
OpenDriveLab/ViDAR
[CVPR 2024 Highlight] Visual Point Cloud Forecasting
wangxiao5791509/MultiModal_BigModels_Survey
[MIR-2023-Survey] A continuously updated paper list for multi-modal pre-trained big models
showlab/all-in-one
[CVPR2023] All in One: Exploring Unified Video-Language Pre-training
DeepGraphLearning/GearNet
GearNet and Geometric Pretraining Methods for Protein Structure Representation Learning, ICLR'2023 (https://arxiv.org/abs/2203.06125)
mczhuge/Kaleido-BERT
💐Kaleido-BERT: Vision-Language Pre-training on Fashion Domain