A curated list of world model for autonmous driving. Keep updated.
-
2024-Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving
arxiv
Paper -
2023-DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model
arxiv
;Generative AI
Paper, Code -
2023-ViDAR: Visual Point Cloud Forecasting enables Scalable Autonomous Driving
arxiv
;Pre-training
;from Shanghai AI Lab
;NuScenes dataset
Paper -
2023-MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations
arxiv
;Pre-training
;CARLA dataset
Paper -
2023-Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion
ICLR 2024
;Future Prediction
;from Waabi
;NuScenes, KITTI Odemetry, Argoverse2 Lidar datasets
Paper -
2023-Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving
arxiv
;Generative AI, Planning
;NuScenes and Waymo datasets
Paper -
2023-ADriver-I: A General World Model for Autonomous Driving
arxiv
;Generative AI
;NuScenes & one private dataset
Paper -
2023-OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving
arxiv
;Occupancy Future Prediction, Planning
;Occ3D dataset for Occupancy Future Prediction, NuScenes for motion planning
Paper, Code -
2023-GAIA-1: A Generative World Model for Autonomous Driving
arxiv
;Generative AI
;Wayve's private data
PaperRelated papers & tutorials to understand this paper:FDM for video diffusion decoder: Paper, Code
Denoising diffusion tutorials: CVPR 2022 tutorial, class from UC Berkeley, Video
-
2023-DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving
arxiv
;Generative AI
;NuScenes dataset
Paper, Code (To be released soon) -
2023-Neural World Models for Computer Vision 'PhD Thesis';
from Wayve
Paper -
2023-UniWorld: Autonomous Driving Pre-training via World Models
arxiv
;Pre-training
;NuScenes dataset
Paper -
2022-Separating the World and Ego Models for Self-Driving
ICLR 2022 workshop on Generalizable Policy Learning in the Physical World
;from Yann Lecun's Group
Paper, Code -
2022-SEM2: Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Model
NeurIPS 2022 Deep Reinforcement Learning Workshop
;RL
;CARLA dataset
Paper -
2022-MILE: Model-Based Imitation Learning for Urban Driving
NeurIPS 2022
;RL
;from Wayve
Paper, Code -
2022-Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models
NeurIPS 2022
Paper, Code -
2021-FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras
ICCV 2019
;Future Prediction
;from Wayve
;NuScenes, Lyft datasets
Paper, Code -
2021-Learning to drive from a world on rails
CVPR 2021 Oral
;RL
Paper, Project Page, Code -
2019-Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
ICLR 2019
;Future Prediction
;from Yann Lecun's Group
Paper, Code
- 2024-CVPR Workshop, Foundation Models for Autonomous Systems, Challenges, Track 4: Predictive World Model
Challenges
Link
- 2024-Data-Centric Evolution in Autonomous Driving: A Comprehensive Survey of Big
Data System, Data Mining, and Closed-Loop Technologies
arxiv
Paper - 2024-Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities
arxiv
Paper
- 2024-Genie: Generative Interactive Environments
Deepmind
Paper, Website - 2024-Sora
OpenAI
,Generative AI
Link, Technical Report - 2024-LWM: World Model on Million-Length Video And Language With RingAttention
arxiv
;Generative AI
Paper, Code - 2024-WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokens
arxiv
;Generative AI
Paper - 2024-Video prediction models as rewards for reinforcement learning
NeurIPS 2024
Paper, Code - 2023-Temporally Consistent Transformers for Video Generation
ICML 2023
Paper, Code - 2023-Learning to Model the World with Language
arxiv
Paper, Code - 2023-Transformers are sample-efficient world models
ICLR 2023
;RL
Paper, Code - 2023-Gradient-based Planning with World Models
arxiv
;from Yann Lecun's Group
;Planning
; Paper - 2023-World Models via Policy-Guided Trajectory Diffusion
arxiv
;RL
; Paper - 2023-DreamerV3: Mastering diverse domains through world models
arxiv
;RL
; Paper, Code - 2022-Daydreamer: World models for physical robot learning
CoRL 2022
;Robotics
Paper, Code - 2022-Masked World Models for Visual Control
CoRL 2022
;Robotics
Paper, Code - 2022-A Path Towards Autonomous Machine Intelligence
openreview
;from Yann Lecun's Group
;General Roadmap for World Models
; Paper; Slides1, Slides2, Slides3; Videos - 2021-LEXA:Discovering and Achieving Goals via World Models
NeurIPS 2021
; Paper, Website & Code - 2021-DreamerV2: Mastering Atari with Discrete World Models
ICLR 2021
;RL
;from Google & Deepmind
Paper, Code - 2020-Dreamer: Dream to Control: Learning Behaviors by Latent Imagination
ICLR 2020
Paper, Code - 2019-Learning Latent Dynamics for Planning from Pixels
ICML 2019
Paper, Code - 2018-Model-Based Planning with Discrete and Continuous Actions
arxiv
;RL, Planning
;from Yann Lecun's Group
; Paper - 2018-Recurrent world models facilitate policy evolution
NeurIPS 2018
; Paper, Code
- 2023-Occupancy Prediction-Guided Neural Planner for Autonomous Driving
ITSC 2023
;Planning, Neural Predicted-Guided Planning
;Waymo Open Motion dataset
Paper
- Readme template from awesome-radar-perception
- Other related repos: Awesome-World-Model Awesome-World-Models-for-AD World models paper list from Shanghai AI lab