/LLM4Opt

A Collection on Large Language Models for Optimization

LLM4Opt

Collection on Large Language Models for Optimization (LLM4Opt)

🔥 Applying Large language models (LLMs) for diverse optimization tasks (Opt) is an emerging research area. This is a collection of references and papers of LLM4Opt. The Papers are sorted by time (first publicly available).

Any suggestions and pull requests are welcomed!

It is far from a comprehensive list. If you want to update the list:

The sharing principle of these references here is for research. If any authors do not want their paper to be listed here, please feel free to contact us.

monthly number of publications

Overview

Platform

Project Description
EoH (Evolution of Heuristics) optimization, mathematics, machine learning, etc
OpenELM robots, image, programming puzzles, etc

Course

Course Description
2024 Fall, LLM Agents LLM basics and LLM for agents

Tutorial&Workshop

Event Link
NeurIPS 2023 Workshop: Foundation Models for Decision Making Link
AAAI 2024 Workshop: Public Sector LLMs: Algorithmic and Sociotechnical Design Link
GECCO 2024 Workshop: Large Language Models for and with Evolutionary Computation (LLMfwEC) Link
GECCO 2024 Workshop: EGML-EC — 3rd GECCO workshop on Enhancing Generative Machine Learning with Evolutionary Computation (EGML-EC) 2024 Link
GECCO 2024 Tutorial: Using Large Language Models for Evolutionary Search Tutorial Link, Tutorial Report Link
PPSN 2024 Tutorial: Large Language Models as Tools for Metaheuristic Design: Exploring Challenges and Opportunities Link
KDD 2024 Tutorial: NL2Code-Reasoning and Planning with LLM for Code Development Link, Link
ICML 2024 Workshop: AI for Math Link
NeurIPS 2024 Workshop: Multimodal Algorithmic Reasoning (MAR) Link
NeurIPS 2024 Workshop: The 4th Workshop on Mathematical Reasoning and AI Link

|

Competition

Event Link
AAAI 2024 Global Competition on Math Problem Solving and Reasoning Link
ICML 2024 Challenges on Automated Math Reasoning Link

SpecialIssues

Event Link
IEEE TETCI, Special Issue on Neural Architecture Search and Large Machine Learning Models Link
ACM TELO, Special Issue on Integrating Evolutionary Algorithms and Large Language Models Link
IEEE TEVC, Special Issue on Evolutionary Computation Meets Large Language Models link

Papers

Review

Title Publication with Date Code Paper
Evolutionary Computation in the Era of Large Language Model: Survey and Roadmap Arxiv, Jan 2024 [code] paper
A Survey of Neural Code Intelligence: Paradigms, Advances and Beyond Arxiv, Mar. 2024 [code] paper
When Large Language Model Meets Optimization Arxiv, May 2024 [code] paper

Position paper

Title Publication with Date Code Paper
The Era OF Semantic Decoding Arxiv, Mar. 2024 [code] paper
Leveraging Foundational Models for Black-Box Optimization: Benefits, Challenges, and Future Directions ICML 2024, May 2024 [code] paper

Algorithm/Heuristic/Function Search

Title Publication with Date Code Paper
Hypothesis Search: Inductive Reasoning with Language Models Arxiv Sep 2023, ICLR 2024 [code] paper
ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Search Arxiv Oct 2023, ICLR 2024 [code] paper
Algorithm Evolution using Large Language Model Arxiv Nov 2023 code paper
Mathematical discoveries from program search with large language models Nature code paper
Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model ICML 2024 (Oral) code paper
ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution Arxiv Feb 2024 code paper
Discovering More Effective Tensor Network Structure Search Algorithms via Large Language Models (LLMs) [code] paper
AutoSAT: Automatically Optimize SAT Solvers via Large Language Models Arxiv Feb 2024 [code] paper
Large Language Models tO Enhance Bayesian Optimization Arxiv Feb 2024, ICLR 2024 [code] paper
On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks Arxiv Feb 2024 [code] paper
How Can LLM Guide RL? A Value-Based Approach Arxiv Feb 2024 code paper
LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generation Arxiv Mar 2024 code paper
Evolve Cost-aware Acquisition Functions Using Large Language Models PPSN 2024 [code] paper
Benchmarking ChatGPT on Algorithmic Reasoning Arxiv April 2024 [code] paper
How Multimodal Integration Boost the Performance of LLM for Optimization: Case Study on Capacitated Vehicle Routing Problems Arxiv March 2024 [code] paper
LLM-ABR: Designing Adaptive Bitrate Algorithms via Large Language Models Arxiv April 2024 [code] paper
Constrained Neural Networks for Interpretable Heuristic Creation to Optimise Computer Algebra Systems Arxiv April 2024 [code] paper
GLHF: General Learned Evolutionary Algorithm Via Hyper Functions Arxiv [code] paper
tnGPS: Discovering Unknown Tensor Network Structure Search Algorithms via Large Language Models (LLMs) ICML 2024 [code] paper
LLaMEA: A Large Language Model Evolutionary Algorithm for Automatically Generating Metaheuristics Arxiv May 2024 code paper
Understanding the Importance of Evolutionary Search in Automated Heuristic Design with Large Language Models PPSN 2024 code paper
Discovering Preference Optimization Algorithms with and for Large Language Models Arxiv June 2024 code paper
OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code Arxiv May 2024 code paper

LLM as Optimizer

Title Publication with Date Code Paper
Large Language Models as Optimizers Arxiv, Sep 2023 code paper
Large language model for multi-objective evolutionary optimization Arxiv, Oct. 2023 code paper
Large Language Models as Evolutionary Optimizers Arxiv, Oct. 2023 [code] paper
Using Large Language Models for Hyperparameter Optimization NeurIPS 2023 [code] paper
Large Language Models As Evolution Strategies Arxiv, Feb. 2024 [code] paper
Large Language Model-based Evolutionary Optimizer: Reasoning with Elitism Arxiv, Mar. 2024 [code] paper
Large Language Model-Aided Evolutionary Search for Constrained Multiobjective Optimization Arxiv, May 9, 2024 [code] paper
Towards Optimizing with Large Language Model KDD 2024 [code] paper

LLM as/for surrogate model

Title Publication with Date Code Paper
Large Language Models as Surrogate Models in Evolutionary Algorithms: A Preliminary Study Arxiv, June 2024 code paper
Large Language Model-assisted Surrogate Modelling for Engineering Optimization CAI, 2024 [code] paper
LLM Performance Predictors are good initializers for Architecture Search Arxiv, Aug 2024 [code] paper

Code Generation

Title Publication with Date Code Paper
L2MAC: LARGE LANGUAGE MODEL AUTOMATIC COMPUTER FOR EXTENSIVE CODE GENERATION Arxiv Oct 2023, ICLR 2024 [code] paper

Prompt Opt

Title Publication with Date Code Paper
Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers Arxiv Sep 2023, ICLR 2024 [code] paper
PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization Arxiv Oct 2023, ICLR 2024 [code] paper

Machine Learning

Title Publication with Date Code Paper
Evolution through Large Models Arxiv, June 2022 code paper
Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph Arxiv, July 2023, ICLR 2024 [code] paper
Evoprompting: Language models for code-level neural architecture search NeuIPS 2023 [code] paper
Eureka: Human-Level Reward Design via Coding Large Language Models ICLR 2024 code paper
Language Model Decoding as Direct Metrics Optimization Arxiv, Oct 2023, ICLR 2024 [code] paper
Label-free Node Classification on Graphs with Large Language Models (LLMS) Arxiv, Oct 2023, ICLR 2024 [code] paper
L-AutoDA: Leveraging Large Language Models for Automated Decision-based Adversarial Attacks GECCO 2024, Jan 2024 [code] paper
Data-driven Discovery with Large Generative Models Arxiv, Feb. 2024 [code] paper
Large Language Model-driven Meta-structure Discovery in Heterogeneous Information Network Arxiv, Feb. 2024 [code] paper
LLM Guided Evolution-The Automation of Models Advancing Models Arxiv, Mar. 2024 [code] paper
Identify Critical Nodes in Complex Network with Large Language Models Arxiv, Mar. 2024 [code] paper
Evolving Interpretable Visual Classifiers with Large Language Models Arxiv, April 2024 [code] paper

Science

Title Publication with Date Code Paper
Exploring evolution-aware & -free protein language models as protein function predictors NeurIPS 2022 [code] paper
A Prompt-Engineered Large Language Model Arxiv, Jan 2024 [code] paper
LLM-SR: Scientific Equation Discovery via Programming with Large Language Models Arxiv, April 2024 [code] paper
Large Language Model Agent as a Mechanical Designer Arxiv, April 2024 [code] paper
LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery Arxiv, May 2024 code paper
Taming Large Language Model For Conversational Protein Design Openreview, 2024 [code] paper

Industry

Title Publication with Date Code Paper
Large Language Models for Supply Chain Optimization Arxiv, July 2023 [code] paper
How Can Large Language Models Help Humans in Design and Manufacturing Arxiv, July 2023 [code] paper
LLM4EDA: Emerging Progress in Large Language Models for Electronic Design Automation Arxiv, Dec. 2023 [code] paper

Related Collections

Collection Link
Awesome LLM Link
Foundation Models for Combinatorial Optimization Link
LLM for Planning Link
MOO-ML-Papers Link