We provide awesome papers and repos on very comprehensive topics as follows.
CoT / VLM / Quantization / Grounding / Text2IMG&VID / Prompt Engineering / Prompt Tuning / Reasoning / Robot / Agent / Planning / Reinforcement-Learning / Feedback / In-Context-Learning / Few-Shot / Zero-Shot / Instruction Tuning / PEFT / RLHF / RAG / Embodied / VQA / Hallucination / Diffusion / Scaling / Context-Window / WorldModel / Memory / Zero-Shot / RoPE / Speech / Perception / Survey / Segmentation / Learge Action Model / Foundation / RoPE / LoRA / PPO / DPO
We strongly recommend checking our Notion table for an interactive experience.
Number of papers and repos in total: 516
| Category | Title | Links | Date |
|---|---|---|---|
| Zero-shot | Can Foundation Models Perform Zero-Shot Task Speci fication For Robot Manipulation? |
||
| World-model | Leveraging Pre-trained Large Language Models to Co nstruct and Utilize World Models for Model-based Task Planning |
ArXiv | 2023/05/07 |
| World-model | Learning and Leveraging World Models in Visual Rep resentation Learning |
||
| World-model | Language Models Meet World Models | ArXiv | |
| World-model | Learning to Model the World with Language | ArXiv | |
| World-model | Diffusion World Model | ArXiv | |
| World-model | Learning to Model the World with Language | ArXiv | |
| VisualPrompt | Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models |
ArXiv | |
| VisualPrompt | Making Large Multimodal Models Understand Arbitrar y Visual Prompts |
ArXiv | |
| VisualPrompt | What does CLIP know about a red circle? Visual pro mpt engineering for VLMs |
ArXiv | |
| VisualPrompt | MOKA: Open-Vocabulary Robotic Manipulation through Mark-Based Visual Prompting |
ArXiv | |
| VisualPrompt | SoM : Set-of-Mark PromptingUnleashes Extraordinary Visual Grounding in GPT-4V |
ArXiv | |
| VisualPrompt | Set-of-Mark Prompting Unleashes Extraordinary Visu al Grounding in GPT-4V |
ArXiv, GitHub | |
| Video | MA-LMM: Memory-Augmented Large Multimodal Model fo r Long-Term Video Understanding |
ArXiv | |
| ViFM, Video | InternVideo2: Scaling Video Foundation Models for Multimodal Video Understanding |
ArXiv, GitHub | |
| VLM, World-model | Large World Model | ArXiv | |
| VLM, VQA | CogVLM: Visual Expert for Pretrained Language Mode ls |
ArXiv | 2023/11/06 |
| VLM, VQA | Chameleon: Plug-and-Play Compositional Reasoning w ith Large Language Models |
ArXiv | 2023/04/19 |
| VLM, VQA | DeepSeek-VL: Towards Real-World Vision-Language Un derstanding01 |
||
| VLM | PaLM: Scaling Language Modeling with Pathways | ArXiv | 2022/04/05 |
| VLM | ScreenAI: A Vision-Language Model for UI and Infog raphics Understanding |
||
| VLM | MoE-LLaVA: Mixture of Experts for Large Vision-Lan guage Models |
ArXiv, GitHub | |
| VLM | LLaVA-NeXT: Improved reasoning, OCR, and world kno wledge |
GitHub | |
| VLM | Mini-Gemini: Mining the Potential of Multi-modalit y Vision Language Models |
ArXiv | |
| Text-to-Image, World-model | World Model on Million-Length Video And Language W ith RingAttention |
ArXiv | |
| Tex2Img | Be Yourself: Bounded Attention for Multi-Subject T ext-to-Image Generation |
ArXiv | |
| Temporal | Explorative Inbetweening of Time and Space | ArXiv | |
| Survey, Video | Video Understanding with Large Language Models: A Survey |
ArXiv | |
| Survey, VLM | MM-LLMs: Recent Advances in MultiModal Large Langu age Models |
||
| Survey, Training | Understanding LLMs: A Comprehensive Overview from Training to Inference |
ArXiv | |
| Survey, TimeSeries | Large Models for Time Series and Spatio-Temporal D ata: A Survey and Outlook |
||
| Survey | Efficient Large Language Models: A Survey | ArXiv, GitHub | |
| Sora, Text-to-Video | Sora: A Review on Background, Technology, Limitati ons, and Opportunities of Large Vision Models |
||
| Sora, Text-to-Video | Mora: Enabling Generalist Video Generation via A M ulti-Agent Framework |
ArXiv | |
| Segmentation | LISA: Reasoning Segmentation via Large Language Mo del |
ArXiv | |
| Segmentation | GRES: Generalized Referring Expression Segmentatio n |
||
| Segmentation | Generalized Decoding for Pixel, Image, and Languag e |
ArXiv | |
| Segmentation | SEEM: Segment Everything Everywhere All at Once | ArXiv, GitHub | |
| Segmentation | SegGPT: Segmenting Everything In Context | ArXiv | |
| Segmentation | Grounded SAM: Assembling Open-World Models for Div erse Visual Tasks |
ArXiv | |
| Scaling | Leave No Context Behind: Efficient Infinite Contex t Transformers with Infini-attention |
ArXiv | |
| SLM, Scaling | Textbooks Are All You Need | ArXiv | |
| Robot, Zero-shot | BC-Z: Zero-Shot Task Generalization with Robotic I mitation Learning |
ArXiv | |
| Robot, Zero-shot | Universal Manipulation Interface: In-The-Wild Robo t Teaching Without In-The-Wild Robots |
ArXiv | |
| Robot, Zero-shot | Mirage: Cross-Embodiment Zero-Shot Policy Transfer with Cross-Painting |
ArXiv | |
| Robot, Task-Decompose, Zero-shot | Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents |
ArXiv | 2022/01/18 |
| Robot, Task-Decompose | SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning |
ArXiv | 2023/07/12 |
| Robot, TAMP | LLM3:Large Language Model-based Task and Motion Pl anning with Motion Failure Reasoning |
ArXiv | |
| Robot, TAMP | Task and Motion Planning with Large Language Model s for Object Rearrangement |
ArXiv | |
| Robot, Survey | Toward General-Purpose Robots via Foundation Model s: A Survey and Meta-Analysis |
ArXiv | 2023/12/14 |
| Robot, Survey | Language-conditioned Learning for Robotic Manipula tion: A Survey |
ArXiv | 2023/12/17 |
| Robot, Survey | Robot Learning in the Era of Foundation Models: A Survey |
ArXiv | 2023/11/24 |
| Robot, Survey | Real-World Robot Applications of Foundation Models : A Review |
ArXiv | |
| Robot | OK-Robot: What Really Matters in Integrating Open- Knowledge Models for Robotics |
ArXiv | |
| Robot | RoCo: Dialectic Multi-Robot Collaboration with Lar ge Language Models |
ArXiv | |
| Robot | Interactive Language: Talking to Robots in Real Ti me |
ArXiv | |
| Robot | Reflexion: Language Agents with Verbal Reinforceme nt Learning |
ArXiv | 2023/03/20 |
| Robot | Generative Expressive Robot Behaviors using Large Language Models |
ArXiv | |
| Robot | RoboCat: A Self-Improving Generalist Agent for Rob otic Manipulation |
||
| Robot | Introspective Tips: Large Language Model for In-Co ntext Decision Making |
ArXiv | |
| Robot | PIVOT: Iterative Visual Prompting Elicits Actionab le Knowledge for VLMs |
ArXiv | |
| Robot | OCI-Robotics: Object-Centric Instruction Augmentat ion for Robotic Manipulation |
ArXiv | |
| Robot | DeliGrasp: Inferring Object Mass, Friction, and Co mpliance with LLMs for Adaptive and Minimally Deforming Grasp Policies |
ArXiv | |
| Robot | VoxPoser: Composable 3D Value Maps for Robotic Man ipulation with Language Models |
ArXiv | |
| Robot | Creative Robot Tool Use with Large Language Models | ArXiv | |
| Robot | AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers |
ArXiv | |
| RoPE | RoFormer: Enhanced Transformer with Rotary Positio n Embedding |
ArXiv | |
| Resource | [Resource] Paperswithcode | ArXiv | |
| Resource | [Resource] huggingface | ArXiv | |
| Resource | [Resource] dailyarxiv | ArXiv | |
| Resource | [Resource] Connectedpapers | ArXiv | |
| Resource | [Resource] Semanticscholar | ArXiv | |
| Resource | [Resource] AlphaSignal | ArXiv | |
| Resource | [Resource] arxiv-sanity | ArXiv | |
| Reinforcement-Learning, VIMA | FoMo Rewards: Can we cast foundation models as rew ard functions? |
ArXiv | |
| Reinforcement-Learning, Robot | Towards A Unified Agent with Foundation Models | ArXiv | |
| Reinforcement-Learning | Large Language Models Are Semi-Parametric Reinforc ement Learning Agents |
ArXiv | |
| Reinforcement-Learning | RLang: A Declarative Language for Describing Parti al World Knowledge to Reinforcement Learning Agents |
ArXiv | |
| Reasoning, Zero-shot | Large Language Models are Zero-Shot Reasoners | ArXiv | |
| Reasoning, VLM, VQA | MM-REACT: Prompting ChatGPT for Multimodal Reasoni ng and Action |
ArXiv | 2023/03/20 |
| Reasoning, Table | Large Language Models are few(1)-shot Table Reason ers |
ArXiv | |
| Reasoning, Symbolic | Symbol-LLM: Leverage Language Models for Symbolic System in Visual Human Activity Reasoning |
ArXiv | |
| Reasoning, Survey | Reasoning with Language Model Prompting: A Survey | ArXiv | |
| Reasoning, Robot | AlphaBlock: Embodied Finetuning for Vision-Languag e Reasoning in Robot Manipulation |
ArXiv | |
| Reasoning, Reward | LET’S REWARD STEP BY STEP: STEP-LEVEL REWARD MODEL AS THE NAVIGATORS FOR REASONING |
ArXiv | |
| Reasoning, Reinforcement-Learning | ReFT: Reasoning with Reinforced Fine-Tuning | ||
| Reasoning | Selection-Inference: Exploiting Large Language Mod els for Interpretable Logical Reasoning |
ArXiv | |
| Reasoning | ReConcile: Round-Table Conference Improves Reasoni ng via Consensus among Diverse LLMs. |
ArXiv | |
| Reasoning | Self-Discover: Large Language Models Self-Compose Reasoning Structures |
ArXiv | |
| Reasoning | Chain-of-Thought Reasoning Without Prompting | ArXiv | |
| Reasoning | Contrastive Chain-of-Thought Prompting | ArXiv | |
| Reasoning | Rephrase and Respond(RaR) | ||
| Reasoning | Take a Step Back: Evoking Reasoning via Abstractio n in Large Language Models |
ArXiv | |
| Reasoning | STaR: Bootstrapping Reasoning With Reasoning | ArXiv | 2022/05/28 |
| Reasoning | The Impact of Reasoning Step Length on Large Langu age Models |
ArXiv | |
| Reasoning | Beyond Natural Language: LLMs Leveraging Alternati ve Formats for Enhanced Reasoning and Communication |
ArXiv | |
| Reasoning | Large Language Models as General Pattern Machines | ArXiv | |
| RLHF, Reinforcement-Learning, Survey | A Survey of Reinforcement Learning from Human Feed back |
||
| RLHF | Secrets of RLHF in Large Language Models Part II: Reward Modeling |
ArXiv | |
| RAG, Temporal Logics | FreshLLMs: Refreshing Large Language Models with S earch Engine Augmentation |
ArXiv | |
| RAG, Survey | Large Language Models for Information Retrieval: A Survey |
||
| RAG, Survey | Retrieval-Augmented Generation for Large Language | ||
| RAG, Survey | Retrieval-Augmented Generation for Large Language Models: A Survey |
ArXiv | |
| RAG | Training Language Models with Memory Augmentation | ||
| RAG | Self-RAG: Learning to Retrieve, Generate, and Crit ique through Self-Reflection |
||
| RAG | RAG-Fusion: a New Take on Retrieval-Augmented Gene ration |
ArXiv | |
| RAG | RAFT: Adapting Language Model to Domain Specific R AG |
ArXiv | |
| RAG | Adaptive-RAG: Learning to Adapt Retrieval-Augmente d Large Language Models through Question Complexity |
ArXiv | |
| RAG | RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Ca se Study on Agriculture |
ArXiv | |
| RAG | Fine-Tuning or Retrieval? Comparing Knowledge Inje ction in LLMs |
ArXiv | |
| Quantization, Scaling | SliceGPT: Compress Large Language Models by Deleti ng Rows and Columns |
ArXiv | |
| Prompting, Survey | A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications |
ArXiv | |
| Prompting, Robot, Zero-shot | Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning |
ArXiv | |
| Prompting | Contrastive Chain-of-Thought Prompting | ArXiv | |
| PersonalCitation, Robot | Text2Motion: From Natural Language Instructions to Feasible Plans |
ArXiv | |
| Perception, Video, Vision | CLIP4Clip: An Empirical Study of CLIP for End to E nd Video Clip Retrieval |
ArXiv | |
| Perception, Task-Decompose | DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment |
ArXiv | 2023/07/01 |
| Perception, Robot, Segmentation | Language Segment-Anything | ||
| Perception, Robot | LiDAR-LLM: Exploring the Potential of Large Langua ge Models for 3D LiDAR Understanding |
ArXiv | 2023/12/21 |
| Perception, Reasoning, Robot | Reasoning Grasping via Multimodal Large Language M odel |
ArXiv | |
| Perception, Reasoning | DetGPT: Detect What You Need via Reasoning | ArXiv | |
| Perception, Reasoning | Lenna: Language Enhanced Reasoning Detection Assis tant |
ArXiv | |
| Perception | Simple Open-Vocabulary Object Detection with Visio n Transformers |
ArXiv | 2022/05/12 |
| Perception | Grounded Language-Image Pre-training | ArXiv | 2021/12/07 |
| Perception | Grounding DINO: Marrying DINO with Grounded Pre-Tr aining for Open-Set Object Detection |
ArXiv | 2023/03/09 |
| Perception | PointCLIP: Point Cloud Understanding by CLIP | ArXiv | 2021/12/04 |
| Perception | DINO: DETR with Improved DeNoising Anchor Boxes fo r End-to-End Object Detection |
ArXiv | |
| Perception | Recognize Anything: A Strong Image Tagging Model | ArXiv | |
| Perception | Simple Open-Vocabulary Object Detection with Visio n Transformers |
ArXiv | |
| Perception | Sigmoid Loss for Language Image Pre-Training | ArXiv | |
| Package | LlamaIndex | GitHub | |
| Package | LangChain | GitHub | |
| Package | h2oGPT | GitHub | |
| Package | Dify | GitHub | |
| Package | Alpaca-LoRA | GitHub | |
| Package | Promptlayer | GitHub | |
| Package | unsloth | GitHub | |
| Package | Instructor: Structured LLM Outputs | GitHub | |
| PRM | Let's Verify Step by Step | ArXiv | |
| PRM | Let's reward step by step: Step-Level reward model as the Navigators for Reasoning |
ArXiv | |
| PPO, RLHF, Reinforcement-Learning | Secrets of RLHF in Large Language Models Part I: P PO |
ArXiv | 2024/02/01 |
| Open-source, VLM | OpenFlamingo: An Open-Source Framework for Trainin g Large Autoregressive Vision-Language Models |
ArXiv | 2023/08/02 |
| Open-source, SLM | RecurrentGemma: Moving Past Transformers for Effic ient Open Language Models |
ArXiv | |
| Open-source, Perception | Grounding DINO: Marrying DINO with Grounded Pre-Tr aining for Open-Set Object Detection |
ArXiv | |
| Open-source | Gemma: Introducing new state-of-the-art open model s |
ArXiv | |
| Open-source | Mistral 7B | ArXiv | |
| Open-source | Qwen Technical Report | ArXiv | |
| Navigation, Reasoning, Vision | NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models |
ArXiv | |
| Natural-Language-as-Polices, Robot | RT-H: Action Hierarchies Using Language | ArXiv | |
| Multimodal, Robot, VLM | Open-World Object Manipulation using Pre-trained V ision-Language Models |
ArXiv | 2023/03/02 |
| Multimodal, Robot | MOMA-Force: Visual-Force Imitation for Real-World Mobile Manipulation |
ArXiv | 2023/08/07 |
| Multimodal, Robot | Flamingo: a Visual Language Model for Few-Shot Lea rning |
ArXiv | 2022/04/29 |
| Multi-Images, VLM | Mantis: Multi-Image Instruction Tuning | ArXiv, GitHub | |
| MoE | Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity |
ArXiv | |
| MoE | Sparse MoE as the New Dropout: Scaling Dense and S elf-Slimmable Transformers |
ArXiv | |
| Mixtral, MoE | Mixtral of Experts | ArXiv | |
| Memory, Robot | LLM as A Robotic Brain: Unifying Egocentric Memory and Control |
ArXiv | 2023/04/19 |
| Memory, Reinforcement-Learning | Semantic HELM: A Human-Readable Memory for Reinfor cement Learning |
||
| Math, Reasoning | DeepSeekMath: Pushing the Limits of Mathematical R easoning in Open Language Models |
ArXiv, GitHub | |
| Math, PRM | Math-Shepherd: Verify and Reinforce LLMs Step-by-s tep without Human Annotations |
ArXiv | |
| Math | WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct |
ArXiv | |
| Math | Llemma: An Open Language Model For Mathematics | ArXiv | |
| Low-level-action, Robot | SayTap: Language to Quadrupedal Locomotion | ArXiv | 2023/06/13 |
| Low-level-action, Robot | Prompt a Robot to Walk with Large Language Models | ArXiv | 2023/09/18 |
| LoRA, Scaling | LoRA: Low-Rank Adaptation of Large Language Models | ArXiv | |
| LoRA, Scaling | Vera: A General-Purpose Plausibility Estimation Mo del for Commonsense Statements |
ArXiv | |
| LoRA | LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report |
ArXiv | |
| Lab | Tencent AI Lab - AppAgent, WebVoyager | ||
| Lab | DeepWisdom - MetaGPT | ||
| Lab | Reworkd AI - AgentGPT | ||
| Lab | OpenBMB - ChatDev, XAgent, AgentVerse | ||
| Lab | XLANG NLP Lab - OpenAgents | ||
| Lab | Rutgers University, AGI Research - OpenAGI | ||
| Lab | Knowledge Engineering Group (KEG) & Data Mining at Tsinghua University - CogVLM |
||
| Lab | OpenGVLab | GitHub | |
| Lab | Imperial College London - Zeroshot trajectory | ||
| Lab | sensetime | ||
| Lab | tsinghua | ||
| Lab | Fudan NLP Group | ||
| Lab | Penn State University | ||
| LLaVA, VLM | TinyLLaVA: A Framework of Small-scale Large Multim odal Models |
ArXiv | |
| LLaVA, MoE, VLM | MoE-LLaVA: Mixture of Experts for Large Vision-Lan guage Models |
||
| LLaMA, Lightweight, Open-source | MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT |
||
| LLM, Zero-shot | GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Re cognition? |
ArXiv | 2023/11/27 |
| LLM, Temporal Logics | NL2TL: Transforming Natural Languages to Temporal Logics using Large Language Models |
ArXiv | 2023/05/12 |
| LLM, Survey | A Survey of Large Language Models | ArXiv | 2023/03/31 |
| LLM, Spacial | Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoning |
ArXiv | 2023/10/05 |
| LLM, Scaling | BitNet: Scaling 1-bit Transformers for Large Langu age Models |
ArXiv | |
| LLM, Robot, Task-Decompose | Do As I Can, Not As I Say: Grounding Language in R obotic Affordances |
ArXiv | 2022/04/04 |
| LLM, Robot, Survey | Large Language Models for Robotics: A Survey | ArXiv | |
| LLM, Reasoning, Survey | Towards Reasoning in Large Language Models: A Surv ey |
ArXiv | 2022/12/20 |
| LLM, Quantization | The Era of 1-bit LLMs: All Large Language Models a re in 1.58 Bits |
ArXiv | |
| LLM, PersonalCitation, Robot, Zero-shot | Language Models as Zero-Shot Trajectory Generators | ArXiv | |
| LLM, PersonalCitation, Robot | Tree-Planner: Efficient Close-loop Task Planning w ith Large Language Models01 |
||
| LLM, Open-source | A self-hosted, offline, ChatGPT-like chatbot, powe red by Llama 2. 100% private, with no data leaving your device. |
GitHub | |
| LLM, Open-source | OpenFlamingo: An Open-Source Framework for Trainin g Large Autoregressive Vision-Language Models |
ArXiv | 2023/08/02 |
| LLM, Open-source | InstructBLIP: Towards General-purpose Vision-Langu age Models with Instruction Tuning |
ArXiv | 2023/05/11 |
| LLM, Open-source | ChatBridge: Bridging Modalities with Large Languag e Model as a Language Catalyst |
ArXiv | 2023/05/25 |
| LLM, Memory | MemoryBank: Enhancing Large Language Models with L ong-Term Memory |
ArXiv | 2023/05/17 |
| LLM, Leaderboard | LMSYS Chatbot Arena Leaderboard | ||
| LLM | Language Models are Few-Shot Learners | ArXiv | 2020/05/28 |
| Intaractive, OpenGVLab, VLM | InternGPT: Solving Vision-Centric Tasks by Interac ting with ChatGPT Beyond Language |
ArXiv | 2023/05/09 |
| Instruction-Turning, Survey | Is Prompt All You Need? No. A Comprehensive and Br oader View of Instruction Learning |
||
| Instruction-Turning, Survey | Vision-Language Instruction Tuning: A Review and A nalysis |
ArXiv | |
| Instruction-Turning, Survey | A Closer Look at the Limitations of Instruction Tu ning |
ArXiv | |
| Instruction-Turning, Survey | A Survey on Data Selection for LLM Instruction Tun ing |
ArXiv | |
| Instruction-Turning, Self | Self-Instruct: Aligning Language Models with Self- Generated Instructions |
ArXiv | |
| Instruction-Turning, LLM, Zero-shot | Finetuned Language Models Are Zero-Shot Learners | ArXiv | 2021/09/03 |
| Instruction-Turning, LLM, Survey | Instruction Tuning for Large Language Models: A Su rvey |
||
| Instruction-Turning, LLM, PEFT | Visual Instruction Tuning | ArXiv | 2023/04/17 |
| Instruction-Turning, LLM, PEFT | LLaMA-Adapter: Efficient Fine-tuning of Language M odels with Zero-init Attention |
ArXiv | 2023/03/28 |
| Instruction-Turning, LLM | Training language models to follow instructions wi th human feedback |
ArXiv | 2022/03/04 |
| Instruction-Turning, LLM | MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models |
ArXiv | 2023/04/20 |
| Instruction-Turning, LLM | Self-Instruct: Aligning Language Models with Self- Generated Instructions |
ArXiv | 2022/12/20 |
| Instruction-Turning | A Closer Look at the Limitations of Instruction Tu ning |
||
| Instruction-Turning | Exploring Format Consistency for Instruction Tunin g |
||
| Instruction-Turning | Exploring the Benefits of Training Expert Language Models over Instruction Tuning |
ArXiv | 2023/02/06 |
| Instruction-Turning | Tuna: Instruction Tuning using Feedback from Large Language Models |
ArXiv | 2023/03/06 |
| In-Context-Learning, Vision | What Makes Good Examples for Visual In-Context Lea rning? |
||
| In-Context-Learning, Vision | Visual Prompting via Image Inpainting | ArXiv | |
| In-Context-Learning, Video | Prompting Visual-Language Models for Efficient Vid eo Understanding |
||
| In-Context-Learning, VQA | VisualCOMET: Reasoning about the Dynamic Context o f a Still Image |
ArXiv | 2020/04/22 |
| In-Context-Learning, VQA | SINC: Self-Supervised In-Context Learning for Visi on-Language Tasks |
ArXiv | 2023/07/15 |
| In-Context-Learning, Survey | A Survey on In-context Learning | ArXiv | |
| In-Context-Learning, Scaling | Structured Prompting: Scaling In-Context Learning to 1,000 Examples |
ArXiv | 2020/03/06 |
| In-Context-Learning, Scaling | Rethinking the Role of Scale for In-Context Learni ng: An Interpretability-based Case Study at 66 Billion Scale |
ArXiv | 2022/03/06 |
| In-Context-Learning, Reinforcement-Learning | AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents |
ArXiv | |
| In-Context-Learning, Prompt-Tuning | Visual Prompt Tuning | ArXiv | |
| In-Context-Learning, Perception, Vision | Visual In-Context Prompting | ArXiv | |
| In-Context-Learning, Many-Shot, Reasoning | Many-Shot In-Context Learning | ArXiv | |
| In-Context-Learning, Instruction-Turning | In-Context Instruction Learning | ||
| In-Context-Learning | ReAct: Synergizing Reasoning and Acting in Languag e Models |
ArXiv | 2023/03/20 |
| In-Context-Learning | Small Models are Valuable Plug-ins for Large Langu age Models |
ArXiv | 2023/05/15 |
| In-Context-Learning | Generative Agents: Interactive Simulacra of Human Behavior |
ArXiv | 2023/04/07 |
| In-Context-Learning | Beyond the Imitation Game: Quantifying and extrapo lating the capabilities of language models |
ArXiv | 2022/06/09 |
| In-Context-Learning | What does CLIP know about a red circle? Visual pro mpt engineering for VLMs |
ArXiv | |
| In-Context-Learning | Can large language models explore in-context? | ArXiv | |
| Image, LLaMA, Perception | LLaMA-VID: An Image is Worth 2 Tokens in Large Lan guage Models |
ArXiv | |
| Hallucination, Survey | Combating Misinformation in the Age of LLMs: Oppor tunities and Challenges |
ArXiv | |
| Gym, PPO, Reinforcement-Learning, Survey | Can Language Agents Approach the Performance of RL ? An Empirical Study On OpenAI Gym |
ArXiv | |
| Grounding, Reinforcement-Learning | Grounding Large Language Models in Interactive Env ironments with Online Reinforcement Learning |
ArXiv | |
| Grounding, Reasoning | Visually Grounded Reasoning across Languages and C ultures |
ArXiv | |
| Grounding | V-IRL: Grounding Virtual Intelligence in Real Life | ||
| Google, Grounding | GLaMM: Pixel Grounding Large Multimodal Model | ArXiv | |
| Generation, Survey | Advances in 3D Generation: A Survey | ||
| Generation, Robot, Zero-shot | Zero-Shot Robotic Manipulation with Pretrained Ima ge-Editing Diffusion Models |
ArXiv | |
| Generation, Robot, Zero-shot | Towards Generalizable Zero-Shot Manipulationvia Tr anslating Human Interaction Plans |
||
| GPT4V, Robot, VLM | Closed-Loop Open-Vocabulary Mobile Manipulation wi th GPT-4V |
ArXiv | |
| GPT4, LLM | GPT-4 Technical Report | ArXiv | 2023/03/15 |
| GPT4, Instruction-Turning | INSTRUCTION TUNING WITH GPT-4 | ArXiv | |
| GPT4, Gemini, LLM | Gemini vs GPT-4V: A Preliminary Comparison and Com bination of Vision-Language Models Through Qualitative Cases |
ArXiv | 2023/12/22 |
| Foundation, Robot, Survey | Foundation Models in Robotics: Applications, Chall enges, and the Future |
ArXiv | 2023/12/13 |
| Foundation, LLaMA, Vision | VisionLLaMA: A Unified LLaMA Interface for Vision Tasks |
ArXiv | |
| Foundation, LLM, Open-source | Code Llama: Open Foundation Models for Code | ArXiv | |
| Foundation, LLM, Open-source | LLaMA: Open and Efficient Foundation Language Mode ls |
ArXiv | 2023/02/27 |
| Feedback, Robot | Correcting Robot Plans with Natural Language Feedb ack |
ArXiv | |
| Feedback, Robot | Learning to Learn Faster from Human Feedback with Language Model Predictive Control |
ArXiv | |
| Feedback, Robot | REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction |
ArXiv | 2023/06/27 |
| Feedback, In-Context-Learning, Robot | InCoRo: In-Context Learning for Robotics Control w ith Feedback Loops |
ArXiv | |
| Evaluation, LLM, Survey | A Survey on Evaluation of Large Language Models | ArXiv | |
| Evaluation | simple-evals | GitHub | |
| End2End, Multimodal, Robot | VIMA: General Robot Manipulation with Multimodal P rompts |
ArXiv | 2022/10/06 |
| End2End, Multimodal, Robot | PaLM-E: An Embodied Multimodal Language Model | ArXiv | 2023/03/06 |
| End2End, Multimodal, Robot | Physically Grounded Vision-Language Models for Rob otic Manipulation |
ArXiv | 2023/09/05 |
| Enbodied | Embodied Question Answering | ArXiv | |
| Embodied, World-model | Language Models Meet World Models: Embodied Experi ences Enhance Language Models |
||
| Embodied, Robot, Task-Decompose | Embodied Task Planning with Large Language Models | ArXiv | 2023/07/04 |
| Embodied, Robot | Large Language Models as Generalizable Policies fo r Embodied Tasks |
ArXiv | |
| Embodied, Reasoning, Robot | Natural Language as Policies: Reasoning for Coordi nate-Level Embodied Control with LLMs |
ArXiv, GitHub | 2024/03/20 |
| Embodied, LLM, Robot, Survey | The Development of LLMs for Embodied Navigation | ArXiv | 2023/11/01 |
| Driving, Spacial | GPT-Driver: Learning to Drive with GPT | ArXiv | 2023/10/02 |
| Drive, Survey | A Survey on Multimodal Large Language Models for A utonomous Driving |
ArXiv | |
| Distilling, Survey | A Survey on Knowledge Distillation of Large Langua ge Models |
||
| Distilling | Distilling Step-by-Step! Outperforming Larger Lang uage Models with Less Training Data and Smaller Model Sizes01 |
ArXiv | |
| Diffusion, Text-to-Image | Mastering Text-to-Image Diffusion: Recaptioning, P lanning, and Generating with Multimodal LLMs |
ArXiv | |
| Diffusion, Survey | On the Design Fundamentals of Diffusion Models: A Survey |
ArXiv | |
| Diffusion, Robot | 3D Diffusion Policy | ArXiv | |
| Diffusion | A latent text-to-image diffusion model | ||
| Demonstration, GPT4, PersonalCitation, Robot, VLM | GPT-4V(ision) for Robotics: Multimodal Task Planni ng from Human Demonstration |
||
| Datatset, LLM, Survey | A Survey on Data Selection for Language Models | ArXiv | |
| Datatset, Instruction-Turning | REVO-LION: EVALUATING AND REFINING VISION LANGUAGE INSTRUCTION TUNING DATASETS |
||
| Datatset, Instruction-Turning | Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models |
||
| Datatset | PRM800K: A Process Supervision Dataset | GitHub | |
| Data-generation, Robot | GenSim: Generating Robotic Simulation Tasks via La rge Language Models |
ArXiv | 2023/10/02 |
| Data-generation, Robot | RoboGen: Towards Unleashing Infinite Data for Auto mated Robot Learning via Generative Simulation |
ArXiv | 2023/11/02 |
| DPO, PPO, RLHF | A Comprehensive Survey of LLM Alignment Techniques : RLHF, RLAIF, PPO, DPO and More |
ArXiv | |
| DPO | Is DPO Superior to PPO for LLM Alignment? A Compre hensive Study |
ArXiv | |
| Context-Window, Scaling | Infini-gram: Scaling Unbounded n-gram Language Mod els to a Trillion Tokens |
||
| Context-Window, Scaling | LONGNET: Scaling Transformers to 1,000,000,000 Tok ens |
ArXiv | 2023/07/01 |
| Context-Window, Reasoning, RoPE, Scaling | Resonance RoPE: Improving Context Length Generaliz ation of Large Language Models |
ArXiv | |
| Context-Window, LLM, RoPE, Scaling | LongRoPE: Extending LLM Context Window Beyond 2 Mi llion Tokens |
ArXiv | |
| Context-Window, Foundation, Gemini, LLM, Scaling | Gemini 1.5: Unlocking multimodal understanding acr oss millions of tokens of context |
||
| Context-Window, Foundation | Mamba: Linear-Time Sequence Modeling with Selectiv e State Spaces |
ArXiv | |
| Context-Window | RoFormer: Enhanced Transformer with Rotary Positio n Embedding |
ArXiv | |
| Context-Awere, Context-Window | DynaCon: Dynamic Robot Planner with Contextual Awa reness via LLMs |
ArXiv | |
| Computer-Resource, Scaling | FlashAttention: Fast and Memory-Efficient Exact At tention with IO-Awareness |
ArXiv | |
| Compress, Scaling | (Long)LLMLingua: Enhancing Large Language Model In ference via Prompt Compression |
ArXiv | |
| Compress, Quantization, Survey | A Survey on Model Compression for Large Language M odels |
ArXiv | |
| Compress, Prompting | Learning to Compress Prompts with Gist Tokens | ArXiv | |
| Code-as-Policies, VLM, VQA | Visual Programming: Compositional visual reasoning without training |
ArXiv | 2022/11/18 |
| Code-as-Policies, Robot | SMART-LLM: Smart Multi-Agent Robot Task Planning u sing Large Language Models |
ArXiv | 2023/09/18 |
| Code-as-Policies, Robot | RoboScript: Code Generation for Free-Form Manipula tion Tasks across Real and Simulation |
ArXiv | |
| Code-as-Policies, Robot | Creative Robot Tool Use with Large Language Models | ArXiv | |
| Code-as-Policies, Reinforcement-Learning, Reward | Code as Reward: Empowering Reinforcement Learning with VLMs |
ArXiv | |
| Code-as-Policies, Reasoning, VLM, VQA | ViperGPT: Visual Inference via Python Execution fo r Reasoning |
ArXiv | 2023/03/14 |
| Code-as-Policies, Reasoning | Chain of Code: Reasoning with a Language Model-Aug mented Code Emulator |
ArXiv | |
| Code-as-Policies, PersonalCitation, Robot, Zero-shot | Socratic Models: Composing Zero-Shot Multimodal Re asoning with Language |
ArXiv | 2022/04/01 |
| Code-as-Policies, PersonalCitation, Robot, State-Manage | Statler: State-Maintaining Language Models for Emb odied Reasoning |
ArXiv | 2023/06/30 |
| Code-as-Policies, PersonalCitation, Robot | ProgPrompt: Generating Situated Robot Task Plans u sing Large Language Models |
ArXiv | 2022/09/22 |
| Code-as-Policies, PersonalCitation, Robot | RoboCodeX:Multi-modal Code Generation forRobotic B ehavior Synthesis |
ArXiv | |
| Code-as-Policies, PersonalCitation, Robot | RoboGPT: an intelligent agent of making embodied l ong-term decisions for daily instruction tasks |
||
| Code-as-Policies, PersonalCitation, Robot | ChatGPT for Robotics: Design Principles and Model Abilities |
||
| Code-as-Policies, Multimodal, OpenGVLab, PersonalCitation, Robot | Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model |
ArXiv | 2023/05/18 |
| Code-as-Policies, Embodied, PersonalCitation, Robot | Code as Policies: Language Model Programs for Embo died Control |
ArXiv | 2022/09/16 |
| Code-as-Policies, Embodied, PersonalCitation, Reasoning, Robot, Task-Decompose | Inner Monologue: Embodied Reasoning through Planni ng with Language Models |
ArXiv | |
| Code-LLM, Front-End | Design2Code: How Far Are We From Automating Front- End Engineering? |
ArXiv | |
| Code-LLM | StarCoder 2 and The Stack v2: The Next Generation | ||
| Chain-of-Thought, Reasoning, Table | Chain-of-table: Evolving tables in the reasoning c hain for table understanding |
ArXiv | |
| Chain-of-Thought, Reasoning, Survey | Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters |
ArXiv | 2023/12/20 |
| Chain-of-Thought, Reasoning, Survey | A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future |
ArXiv | 2023/09/27 |
| Chain-of-Thought, Reasoning | Chain-of-Thought Prompting Elicits Reasoning in La rge Language Models |
ArXiv | 2022/01/28 |
| Chain-of-Thought, Reasoning | Tree of Thoughts: Deliberate Problem Solving with Large Language Models |
ArXiv | 2023/05/17 |
| Chain-of-Thought, Reasoning | Multimodal Chain-of-Thought Reasoning in Language Models |
ArXiv | 2023/02/02 |
| Chain-of-Thought, Reasoning | Verify-and-Edit: A Knowledge-Enhanced Chain-of-Tho ught Framework |
ArXiv | 2023/05/05 |
| Chain-of-Thought, Reasoning | Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding |
ArXiv | 2023/07/28 |
| Chain-of-Thought, Reasoning | Rethinking with Retrieval: Faithful Large Language Model Inference |
ArXiv | 2022/12/31 |
| Chain-of-Thought, Reasoning | Self-Consistency Improves Chain of Thought Reasoni ng in Language Models |
ArXiv | 2022/03/21 |
| Chain-of-Thought, Reasoning | Chain-of-Thought Hub: A Continuous Effort to Measu re Large Language Models' Reasoning Performance |
ArXiv | 2023/05/26 |
| Chain-of-Thought, Reasoning | Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation |
ArXiv | |
| Chain-of-Thought, Prompting | Chain-of-Thought Reasoning Without Prompting | ArXiv | |
| Chain-of-Thought, Planning, Reasoning | SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning |
ArXiv | 2023/08/01 |
| Chain-of-Thought, In-Context-Learning, Self | Measuring and Narrowing the Compositionality Gap i n Language Models |
ArXiv | 2022/10/07 |
| Chain-of-Thought, In-Context-Learning, Self | Self-Polish: Enhance Reasoning in Large Language M odels via Problem Refinement |
ArXiv | 2023/05/23 |
| Chain-of-Thought, In-Context-Learning | Chain-of-Table: Evolving Tables in the Reasoning C hain for Table Understanding |
ArXiv | |
| Chain-of-Thought, In-Context-Learning | Self-Refine: Iterative Refinement with Self-Feedba ck |
ArXiv | 2023/03/30 |
| Chain-of-Thought, In-Context-Learning | Plan-and-Solve Prompting: Improving Zero-Shot Chai n-of-Thought Reasoning by Large Language Models |
ArXiv | 2023/05/06 |
| Chain-of-Thought, In-Context-Learning | PAL: Program-aided Language Models | ArXiv | 2022/11/18 |
| Chain-of-Thought, In-Context-Learning | Reasoning with Language Model is Planning with Wor ld Model |
ArXiv | 2023/05/24 |
| Chain-of-Thought, In-Context-Learning | Least-to-Most Prompting Enables Complex Reasoning in Large Language Models |
ArXiv | 2022/05/21 |
| Chain-of-Thought, In-Context-Learning | Complexity-Based Prompting for Multi-Step Reasonin g |
ArXiv | 2022/10/03 |
| Chain-of-Thought, In-Context-Learning | Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations |
ArXiv | 2022/05/24 |
| Chain-of-Thought, In-Context-Learning | Algorithm of Thoughts: Enhancing Exploration of Id eas in Large Language Models |
ArXiv | 2023/08/20 |
| Chain-of-Thought, GPT4, Reasoning, Robot | Look Before You Leap: Unveiling the Power ofGPT-4V in Robotic Vision-Language Planning |
ArXiv | 2023/11/29 |
| Chain-of-Thought, Embodied, Robot | EgoCOT: Embodied Chain-of-Thought Dataset for Visi on Language Pre-training |
ArXiv | |
| Chain-of-Thought, Embodied, PersonalCitation, Robot, Task-Decompose | EmbodiedGPT: Vision-Language Pre-Training via Embo died Chain of Thought |
ArXiv | 2023/05/24 |
| Chain-of-Thought, Code-as-Policies, PersonalCitation, Robot | Demo2Code: From Summarizing Demonstrations to Synt hesizing Code via Extended Chain-of-Thought |
ArXiv | |
| Chain-of-Thought, Code-as-Policies | Chain of Code: Reasoning with a Language Model-Aug mented Code Emulator |
ArXiv | |
| Caption, Video | PLLaVA : Parameter-free LLaVA Extension from Image s to Videos for Video Dense Captioning |
ArXiv | |
| Caption, VLM, VQA | Caption Anything: Interactive Image Description wi th Diverse Multimodal Controls |
ArXiv | 2023/05/04 |
| CRAG, RAG | Corrective Retrieval Augmented Generation | ArXiv | |
| Brain, Instruction-Turning | Instruction-tuning Aligns LLMs to the Human Brain | ArXiv | |
| Brain, Conscious | Could a Large Language Model be Conscious? | ||
| Brain | LLM-BRAIn: AI-driven Fast Generation of Robot Beha viour Tree based on Large Language Model |
ArXiv | |
| Brain | A Neuro-Mimetic Realization of the Common Model of Cognition via Hebbian Learning and Free Energy Minimization |
||
| Benchmark, Sora, Text-to-Video | LIDA: A Tool for Automatic Generation of Grammar-A gnostic Visualizations and Infographics using Large Language Models01 |
||
| Benchmark, In-Context-Learning | ARB: Advanced Reasoning Benchmark for Large Langua ge Models |
ArXiv | 2023/07/25 |
| Benchmark, In-Context-Learning | PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change |
ArXiv | 2022/06/21 |
| Benchmark, GPT4 | Sparks of Artificial General Intelligence: Early e xperiments with GPT-4 |
||
| Awesome Repo, VLM | awesome-vlm-architectures | GitHub | |
| Awesome Repo, Survey | LLMSurvey | GitHub | |
| Awesome Repo, Robot | Awesome-LLM-Robotics | GitHub | |
| Awesome Repo, Reasoning | Awesome LLM Reasoning | GitHub | |
| Awesome Repo, Reasoning | Awesome-Reasoning-Foundation-Models | GitHub | |
| Awesome Repo, RLHF, Reinforcement-Learning | Awesome RLHF (RL with Human Feedback) | GitHub | |
| Awesome Repo, Perception, VLM | Awesome Vision-Language Navigation | GitHub | |
| Awesome Repo, Package | Awesome LLMOps | GitHub | |
| Awesome Repo, Multimodal | Awesome-Multimodal-LLM | GitHub | |
| Awesome Repo, Multimodal | Awesome-Multimodal-Large-Language-Models | GitHub | |
| Awesome Repo, Math, Science | Awesome Scientific Language Models | GitHub | |
| Awesome Repo, LLM, Vision | LLM-in-Vision | GitHub | |
| Awesome Repo, LLM, VLM | Multimodal & Large Language Models | GitHub | |
| Awesome Repo, LLM, Survey | Awesome-LLM-Survey | GitHub | |
| Awesome Repo, LLM, Robot | Everything-LLMs-And-Robotics | GitHub | |
| Awesome Repo, LLM, Leaderboard | LLM-Leaderboard | GitHub | |
| Awesome Repo, LLM | Awesome-LLM | GitHub | |
| Awesome Repo, Korean | awesome-korean-llm | GitHub | |
| Awesome Repo, Japanese, LLM | 日本語LLMまとめ | GitHub | |
| Awesome Repo, In-Context-Learning | Paper List for In-context Learning | GitHub | |
| Awesome Repo, IROS, Robot | IROS2023PaperList | GitHub | |
| Awesome Repo, Hallucination, Survey | A Survey on Hallucination in Large Language Models : Principles, Taxonomy, Challenges, and Open Questions |
ArXiv, GitHub | |
| Awesome Repo, Embodied | Awesome Embodied Vision | GitHub | |
| Awesome Repo, Diffusion | Awesome-Diffusion-Models | GitHub | |
| Awesome Repo, Compress | Awesome LLM Compression | GitHub | |
| Awesome Repo, Chinese | Awesome-Chinese-LLM | GitHub | |
| Awesome Repo, Chain-of-Thought | Chain-of-ThoughtsPapers | GitHub | |
| Automate, Prompting | Large Language Models Are Human-Level Prompt Engin eers |
ArXiv | 2022/11/03 |
| Automate, Chain-of-Thought, Reasoning | Automatic Chain of Thought Prompting in Large Lang uage Models |
ArXiv | 2022/10/07 |
| Audio2Video, Diffusion, Generation, Video | EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions |
ArXiv | |
| Audio | Robust Speech Recognition via Large-Scale Weak Sup ervision |
||
| Apple, VLM | MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training |
ArXiv | |
| Apple, VLM | Guiding Instruction-based Image Editing via Multim odal Large Language Models |
ArXiv | |
| Apple, Robot | Large Language Models as Generalizable Policies fo r Embodied Tasks |
ArXiv | |
| Apple, LLM, Open-source | OpenELM: An Efficient Language Model Family with O pen Training and Inference Framework |
ArXiv | |
| Apple, LLM | ReALM: Reference Resolution As Language Modeling | ArXiv | |
| Apple, LLM | LLM in a flash: Efficient Large Language Model Inf erence with Limited Memory |
ArXiv | |
| Apple, In-Context-Learning, Perception | SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding |
ArXiv | |
| Apple, Code-as-Policies, Robot | Executable Code Actions Elicit Better LLM Agents | ArXiv | |
| Apple | Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models |
ArXiv | |
| Apple | Ferret-UI: Grounded Mobile UI Understanding with M ultimodal LLMs |
ArXiv | |
| Apple | Ferret: Refer and Ground Anything Anywhere at Any Granularity |
ArXiv | |
| Anything, LLM, Open-source, Perception, Segmentation | Segment Anything | ArXiv | 2023/04/05 |
| Anything, Depth | Depth Anything: Unleashing the Power of Large-Scal e Unlabeled Data |
ArXiv | |
| Anything, Caption, Perception, Segmentation | Segment and Caption Anything | ArXiv | |
| Anything, CLIP, Perception | SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding |
ArXiv | |
| Agent-Project, Code-LLM | open-interpreter | GitHub | |
| Agent, Web | WebLINX: Real-World Website Navigation with Multi- Turn Dialogue |
ArXiv | |
| Agent, Web | WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models |
ArXiv | |
| Agent, Web | OmniACT: A Dataset and Benchmark for Enabling Mult imodal Generalist Autonomous Agents for Desktop and Web |
ArXiv | |
| Agent, Web | OS-Copilot: Towards Generalist Computer Agents wit h Self-Improvement |
ArXiv | |
| Agent, Video-for-Agent | Video as the New Language for Real-World Decision Making |
||
| Agent, VLM | AssistGPT: A General Multi-modal Assistant that ca n Plan, Execute, Inspect, and Learn |
ArXiv | |
| Agent, Tool | Gorilla: Large Language Model Connected with Massi ve APIs |
ArXiv | |
| Agent, Tool | ToolLLM: Facilitating Large Language Models to Mas ter 16000+ Real-world APIs |
ArXiv | |
| Agent, Survey | A Survey on Large Language Model based Autonomous Agents |
ArXiv | 2023/08/22 |
| Agent, Survey | The Rise and Potential of Large Language Model Bas ed Agents: A Survey |
ArXiv | 2023/09/14 |
| Agent, Survey | Agent AI: Surveying the Horizons of Multimodal Int eraction |
ArXiv | |
| Agent, Survey | Large Multimodal Agents: A Survey | ArXiv | |
| Agent, Soft-Dev | Communicative Agents for Software Development | GitHub | |
| Agent, Soft-Dev | MetaGPT: Meta Programming for A Multi-Agent Collab orative Framework |
ArXiv | |
| Agent, Robot, Survey | A Survey on LLM-based Autonomous Agents | GitHub | |
| Agent, Reinforcement-Learning, Reward | Reward Design with Language Models | ArXiv | 2023/02/27 |
| Agent, Reinforcement-Learning, Reward | EAGER: Asking and Answering Questions for Automati c Reward Shaping in Language-guided RL |
ArXiv | 2022/06/20 |
| Agent, Reinforcement-Learning, Reward | Text2Reward: Automated Dense Reward Function Gener ation for Reinforcement Learning |
ArXiv | 2023/09/20 |
| Agent, Reinforcement-Learning | Eureka: Human-Level Reward Design via Coding Large Language Models |
ArXiv | 2023/10/19 |
| Agent, Reinforcement-Learning | Language to Rewards for Robotic Skill Synthesis | ArXiv | 2023/06/14 |
| Agent, Reinforcement-Learning | Language Instructed Reinforcement Learning for Hum an-AI Coordination |
ArXiv | 2023/04/13 |
| Agent, Reinforcement-Learning | Guiding Pretraining in Reinforcement Learning with Large Language Models |
ArXiv | 2023/02/13 |
| Agent, Reinforcement-Learning | STARLING: SELF-SUPERVISED TRAINING OF TEXTBASED RE INFORCEMENT LEARNING AGENT WITH LARGE LANGUAGE MODELS |
||
| Agent, Reasoning, Zero-shot | Agent Instructs Large Language Models to be Genera l Zero-Shot Reasoners |
ArXiv | 2023/10/05 |
| Agent, Reasoning | Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoning |
ArXiv | |
| Agent, Reasoning | AGENT INSTRUCTS LARGE LANGUAGE MODELS TO BE GENERA L ZERO-SHOT REASONERS |
ArXiv | |
| Agent, Multimodal, Robot | A Generalist Agent | ArXiv | 2022/05/12 |
| Agent, Multi | War and Peace (WarAgent): Large Language Model-bas ed Multi-Agent Simulation of World Wars |
ArXiv | |
| Agent, MobileApp | You Only Look at Screens: Multimodal Chain-of-Acti on Agents |
ArXiv, GitHub | |
| Agent, Minecraft, Reinforcement-Learning | RLAdapter: Bridging Large Language Models to Reinf orcement Learning in Open Worlds |
||
| Agent, Minecraft | Voyager: An Open-Ended Embodied Agent with Large L anguage Models |
ArXiv | 2023/05/25 |
| Agent, Minecraft | Describe, Explain, Plan and Select: Interactive Pl anning with Large Language Models Enables Open-World Multi-Task Agents |
ArXiv | 2023/02/03 |
| Agent, Minecraft | LARP: Language-Agent Role Play for Open-World Game s |
ArXiv | |
| Agent, Minecraft | Steve-Eye: Equipping LLM-based Embodied Agents wit h Visual Perception in Open Worlds |
ArXiv | |
| Agent, Minecraft | S-Agents: Self-organizing Agents in Open-ended Env ironment |
ArXiv | |
| Agent, Minecraft | Ghost in the Minecraft: Generally Capable Agents f or Open-World Environments via Large Language Models with Text-based Knowledge and Memory01 |
ArXiv | |
| Agent, Memory, RAG, Robot | RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agents |
ArXiv | 2024/02/06 |
| Agent, Memory, Minecraft | JARVIS-1: Open-World Multi-task Agents with Memory -Augmented Multimodal Language Models |
ArXiv | 2023/11/10 |
| Agent, LLM, Planning | LLM-Planner: Few-Shot Grounded Planning for Embodi ed Agents with Large Language Models |
ArXiv | |
| Agent, Instruction-Turning | AgentTuning: Enabling Generalized Agent Abilities For LLMs |
ArXiv | |
| Agent, Game | LEARNING EMBODIED VISION-LANGUAGE PRO- GRAMMING FR OM INSTRUCTION, EXPLORATION, AND ENVIRONMENTAL FEEDBACK |
ArXiv | |
| Agent, GUI, Web | "What’s important here?": Opportunities and Challe nges of Using LLMs in Retrieving Informatio from Web Interfaces |
ArXiv | |
| Agent, GUI, MobileApp | Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception |
||
| Agent, GUI, MobileApp | AppAgent: Multimodal Agents as Smartphone Users | ArXiv | |
| Agent, GUI, MobileApp | You Only Look at Screens: Multimodal Chain-of-Acti on Agents |
||
| Agent, GUI | CogAgent: A Visual Language Model for GUI Agents | ArXiv | |
| Agent, GUI | ScreenAgent: A Computer Control Agent Driven by Vi sual Language Large Model |
GitHub | |
| Agent, GUI | SeeClick: Harnessing GUI Grounding for Advanced Vi sual GUI Agents |
ArXiv | |
| Agent, GPT4, Web | GPT-4V(ision) is a Generalist Web Agent, if Ground ed |
ArXiv | |
| Agent, Feedback, Reinforcement-Learning, Robot | Accelerating Reinforcement Learning of Robotic Man ipulations via Feedback from Large Language Models |
ArXiv | 2023/11/04 |
| Agent, Feedback, Reinforcement-Learning | AdaRefiner: Refining Decisions of Language Models with Adaptive Feedback |
ArXiv | 2023/09/29 |
| Agent, End2End, Game, Robot | An Interactive Agent Foundation Model | ArXiv | |
| Agent, Embodied, Survey | Application of Pretrained Large Language Models in Embodied Artificial Intelligence |
ArXiv | |
| Agent, Embodied, Robot | AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents |
ArXiv | |
| Agent, Embodied, Robot | OPEx: A Component-Wise Analysis of LLM-Centric Age nts in Embodied Instruction Following |
ArXiv | |
| Agent, Embodied | OpenAgents: An Open Platform for Language Agents i n the Wild |
ArXiv, GitHub | |
| Agent, Embodied | LLM-Planner: Few-Shot Grounded Planning for Embodi ed Agents with Large Language Models |
ArXiv | |
| Agent, Embodied | Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld |
ArXiv | |
| Agent, Embodied | Octopus: Embodied Vision-Language Programmer from Environmental Feedback |
||
| Agent, Embodied | Embodied Task Planning with Large Language Models | ArXiv | |
| Agent, Diffusion, Speech | NaturalSpeech 3: Zero-Shot Speech Synthesis with F actorized Codec and Diffusion Models |
ArXiv | |
| Agent, Code-as-Policies | Executable Code Actions Elicit Better LLM Agents | ArXiv | 2024/01/24 |
| Agent, Code-LLM, Code-as-Policies, Survey | If LLM Is the Wizard, Then Code Is the Wand: A Sur vey on How Code Empowers Large Language Models to Serve as Intelligent Agents |
ArXiv | |
| Agent, Code-LLM | TaskWeaver: A Code-First Agent Framework | ||
| Agent, Blog | LLM Powered Autonomous Agents | ArXiv | |
| Agent, Awesome Repo, LLM | Awesome-Embodied-Agent-with-LLMs | GitHub | |
| Agent, Awesome Repo, LLM | CoALA: Awesome Language Agents | ArXiv, GitHub | |
| Agent, Awesome Repo, Embodied, Grounding | XLang Paper Reading | GitHub | |
| Agent, Awesome Repo | Awesome AI Agents | GitHub | |
| Agent, Awesome Repo | Autonomous Agents | GitHub | |
| Agent, Awesome Repo | Awesome-Papers-Autonomous-Agent | GitHub | |
| Agent, Awesome Repo | Awesome Large Multimodal Agents | GitHub | |
| Agent, Awesome Repo | LLM Agents Papers | GitHub | |
| Agent, Awesome Repo | Awesome LLM-Powered Agent | GitHub | |
| Agent | XAgent: An Autonomous Agent for Complex Task Solvi ng |
||
| Agent | LLM-Powered Hierarchical Language Agent for Real-t ime Human-AI Coordination |
ArXiv | |
| Agent | AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors |
ArXiv | |
| Agent | Agents: An Open-source Framework for Autonomous La nguage Agents |
ArXiv, GitHub | |
| Agent | AutoAgents: A Framework for Automatic Agent Genera tion |
GitHub | |
| Agent | DSPy: Compiling Declarative Language Model Calls i nto Self-Improving Pipelines |
ArXiv | |
| Agent | AutoGen: Enabling Next-Gen LLM Applications via Mu lti-Agent Conversation |
ArXiv | |
| Agent | CAMEL: Communicative Agents for “Mind” Exploration of Large Language Model Society |
ArXiv | |
| Agent | XAgent: An Autonomous Agent for Complex Task Solvi ng |
ArXiv | |
| Agent | Generative Agents: Interactive Simulacra of Human Behavior |
ArXiv | |
| Agent | LLM+P: Empowering Large Language Models with Optim al Planning Proficiency |
ArXiv | 2023/04/22 |
| Agent | AgentSims: An Open-Source Sandbox for Large Langua ge Model Evaluation |
ArXiv | 2023/08/08 |
| Agent | Agents: An Open-source Framework for Autonomous La nguage Agents |
ArXiv | |
| Agent | MindAgent: Emergent Gaming Interaction | ArXiv | |
| Agent | InfiAgent: A Multi-Tool Agent for AI Operating Sys tems |
||
| Agent | Predictive Minds: LLMs As Atypical Active Inferenc e Agents |
||
| Agent | swarms | GitHub | |
| Agent | ScreenAgent: A Vision Language Model-driven Comput er Control Agent |
ArXiv | |
| Agent | AssistGPT: A General Multi-modal Assistant that ca n Plan, Execute, Inspect, and Learn |
ArXiv | |
| Agent | PromptAgent: Strategic Planning with Language Mode ls Enables Expert-level Prompt Optimization |
ArXiv | |
| Agent | Cognitive Architectures for Language Agents | ArXiv | |
| Agent | AIOS: LLM Agent Operating System | ArXiv | |
| Agent | LLM as OS, Agents as Apps: Envisioning AIOS, Agent s and the AIOS-Agent Ecosystem |
ArXiv | |
| Agent | Towards General Computer Control: A Multimodal Age nt for Red Dead Redemption II as a Case Study |
||
| Affordance, Segmentation | ManipVQA: Injecting Robotic Affordance and Physica lly Grounded Information into Multi-Modal Large Language Models |
ArXiv | |
| Action-Model, Agent, LAM | LaVague | GitHub | |
| Action-Generation, Generation, Prompting | Prompt a Robot to Walk with Large Language Models | ArXiv | |
| APIs, Agent, Tool | Gorilla: Large Language Model Connected with Massi ve APIs |
ArXiv | |
| AGI, Survey | Levels of AGI: Operationalizing Progress on the Pa th to AGI |
ArXiv | |
| AGI, Brain | When Brain-inspired AI Meets AGI | ArXiv | |
| AGI, Brain | Divergences between Language Models and Human Brai ns |
ArXiv | |
| AGI, Awesome Repo, Survey | Awesome-LLM-Papers-Toward-AGI | GitHub | |
| AGI, Agent | OpenAGI: When LLM Meets Domain Experts | ||
| 3D, Open-source, Perception, Robot | 3D-LLM: Injecting the 3D World into Large Language Models |
ArXiv | 2023/07/24 |
| 3D, GPT4, VLM | GPT-4V(ision) is a Human-Aligned Evaluator for Tex t-to-3D Generation |
ArXiv | |
| ChatEval: Towards Better LLM-based Evaluators thro ugh Multi-Agent Debate |
ArXiv | 2023/08/14 |