Website | Documentation | Examples | GUI | Changelog | Contributing
LLM4AD is an open-source Python-based Platform leveraging Large Language Models (LLMs) for Automatic Algorithm Design (AD). Please refer to the paper [LLM4AD] for detailed information, including the overview, methodology, and benchmark results.
LLM4AD is accomplished with Documents and Examples materials to support users and developers to easily test, build and deploy their own LLM4AD applications and conduct research.
LLM4AD was originally developed for optimisation tasks. The framework is versatile enough to be used in other areas, including machine learning, science discovery, game theory and engineering design.
For more information, see the contact list
-
2026.02 🎉🎉 New Champion! Our LLM4AD system won the CVRPLib BKS competition and established 51 new Best Known Solutions on large-scale CVRP benchmarks.
-
2026.01 🎉🎉 Our Survey Paper "A Systematic Survey on Large Language Models for Algorithm Design" has been accepted by ACM Computing Surveys ! A Rep for the Survey can be found here
-
2025.6 🎉🎉 We're excited to share that EoH recently set a New World Record in Circle Packing Problem, achieving a score of 2.63594 for 26 circles ! Results here
-
2025.6 🎉🎉 LLM4AD at CEC 2025 Tutorial !
-
2025.3 🎉🎉 LLM4AD at EMO 2025 Tutorial !
-
2024.12 🎉🎉 LLM4AD paper Released “LLM4AD: A Platform for Algorithm Design with Large Language Model" !
-
2024.11 🎉🎉 LLM4AD v1.0 Released !
-
2024.10 🎉🎉 Survey Paper “A Systematic Survey on Large Language Models for Algorithm Design” is online !
| Feature | Support / To be supported |
|---|---|
| Unified Interfaces for methods, tasks, LLMs | 🔥Support |
| Evaluation acceleration: multiprocessing evaluation | 🔥Support |
| Process evaluation: separate process and optional wall-clock timeout; see security limits | 🔥Support |
| Logs: local logs, Wandb and Tensorboard support | 🔥Support |
| GUI: methods selection, tasks selection, convergence, best algorithm, ... | 🔥Support |
| Checkpoint warm starts | Method-dependent support and limits |
| Support other programming languages | 🚀Coming soon |
| More search methods | 🚀Coming soon |
| More task examples | 🚀Coming soon |
Important
The core supports Python 3.9 through 3.13. Optional features can have narrower dependency limits.
LLM4AD uses a small core installation. GUI, local-model, logging and specialized task dependencies are optional extras. The core targets Python 3.9–3.13; optional features have separate compatibility limits.
Install from PyPI
$ pip install llm4adTo install every optional dependency from PyPI:
$ pip install "llm4ad[all]"Install from source
Run these commands from the repository root, where setup.py is located:
$ cd LLM4AD
$ pip install .To install every optional dependency from a cloned repository:
$ cd LLM4AD
$ pip install ".[all]"Use llm4ad[extra-name] when installing from PyPI, and .[extra-name] when installing from a local clone. The all extra installs every optional dependency, including local-vLLM dependencies. If your platform cannot install vLLM, install only the smaller extras you need from the tables below.
Method optional dependencies
| Method | Extra packages |
|---|---|
| FunSearch | Local: pip install ".[funsearch]"PyPI: pip install "llm4ad[funsearch]"scipy |
| MOEAD / NSGA2 | Local: pip install ".[multi-objective]"PyPI: pip install "llm4ad[multi-objective]"pymoo |
| MEoH | Local: pip install ".[meoh]"PyPI: pip install "llm4ad[meoh]"pymoo, tree-sitter, tree-sitter-python, plus CodeBLEU 0.7.1 (see below) |
| PartEvo | Local: pip install ".[partevo]"PyPI: pip install "llm4ad[partevo]"tree-sitter, tree-sitter-python, scikit-learn, seaborn, torch, transformers, plus CodeBLEU 0.7.1 (see below) |
| LLaMEA | Local: pip install ".[llamea]"PyPI: pip install "llm4ad[llamea]"llamea, ConfigSpace |
MEoH and PartEvo compare programs with CodeBLEU. The PyPI release codebleu==0.7.0 requires tree-sitter<0.23 and cannot load the Python grammar, so install the compatible 0.7.1 build from GitHub after the extra:
python -m pip install "codebleu @ https://github.com/k4black/codebleu/archive/b0edb622f6a52fe9d1edc407be5061d3e1462a7f.zip" "tree-sitter==0.23.2" "tree-sitter-python==0.23.0"MEoH and PartEvo (with 'ast' features) check this when they are created and stop with this command if CodeBLEU is missing or incompatible.
Task optional dependencies
| Task | Extra packages |
|---|---|
| Machine-learning control tasks | Local: pip install ".[machine-learning]"PyPI: pip install "llm4ad[machine-learning]"gymnasium[box2d] |
| Science-discovery tasks | Local: pip install ".[science-discovery]"PyPI: pip install "llm4ad[science-discovery]"pandas, scipy, sympy |
| Pymoo MOEA/D task | Local: pip install ".[pymoo-task]"PyPI: pip install "llm4ad[pymoo-task]"pymoo |
| Co-Bench tasks | Local: pip install ".[co-bench]"PyPI: pip install "llm4ad[co-bench]"datasets, huggingface_hub, httpx, httpcore, networkx, scipy |
| TSP GLS 2O task | Local: pip install ".[tsp-gls]"PyPI: pip install "llm4ad[tsp-gls]"numba, scipy |
Tool optional dependencies
| Tool | Extra install |
|---|---|
| TensorBoard profiler | Local: pip install ".[tensorboard]"PyPI: pip install "llm4ad[tensorboard]"torch, tensorboard |
| Weights & Biases profiler | Local: pip install ".[wandb]"PyPI: pip install "llm4ad[wandb]"wandb |
| OpenAI backend | Local: pip install ".[openai]"PyPI: pip install "llm4ad[openai]"openai |
| Local Ollama backend | Local: pip install ".[local-ollama]"PyPI: pip install "llm4ad[local-ollama]"langchain-ollama |
| Local vLLM backend | Local: pip install ".[local-vllm]"PyPI: pip install "llm4ad[local-vllm]"requests, psutil, torch, flask, flask-cors, transformers, vllm |
This is version 1.20.0. See CHANGELOG, contributing, and execution limits.
python -m unittest discover -s tests -v
python -m pip install ".[test,docs]"
python -m coverage run -m unittest discover -s tests
python -m coverage combine
python -m coverage json
python -m sphinx -W -b html docs docs/_build/htmlAPI reference, developer guides, record/replay and local open-weight example, and task catalogue/counting rules are versioned with the code. Response replay supports deterministic serial evaluation; it does not guarantee identical new generations from a remote or local model. Process separation and timeouts are not an adversarial-code sandbox.
Note
Configure your LLM api before running the script. For example:
- Set
host: 'api.deepseek.com' - Set
key: 'your api key' - Set
model: 'deepseek-chat'
from llm4ad.task.optimization.online_bin_packing import OBPEvaluation
from llm4ad.tools.llm.llm_api_https import HttpsApi
from llm4ad.method.eoh import EoH, EoHProfiler
if __name__ == '__main__':
llm = HttpsApi(
host='xxx', # your host endpoint, e.g., api.openai.com, api.deepseek.com
key='sk-xxx', # your key, e.g., sk-xxxxxxxxxx
model='xxx', # your llm, e.g., gpt-3.5-turbo, deepseek-chat
timeout=20
)
task = OBPEvaluation()
method = EoH(
llm=llm,
profiler=EoHProfiler(log_dir='logs/eoh', log_style='simple'),
evaluation=task,
max_sample_nums=20,
max_generations=10,
pop_size=4,
num_samplers=1,
num_evaluators=1,
debug_mode=False
)
method.run()Check Documents for more tasks and examples
Important
Install GUI dependencies with pip install ".[gui]" from a source checkout. For machine-learning tasks, use pip install ".[gui,machine-learning]". The launcher currently runs from the checkout, not from the wheel.
$ cd GUI
$ python run_gui.pyCheck GUI Introduction for more information
| Area | Algorithm Task | Paper |
|---|---|---|
| Optimization | [Online Bin Packing, Constructive heuristic] | paper |
| Traveling Salesman Problem (TSP), Construct heuristic | paper | |
| Traveling Salesman Problem (TSP), Guided local search | paper | |
| 1-dimensional Bin Packing (BP1D), Construct heuristic | paper | |
| 2-dimensional Bin Packing (BP2D), Construct heuristic | paper | |
| Capacitated Facility Location Problem (CFLP), Construct heuristic | paper | |
| Knapsack Problem (KP), Construct heuristic | paper | |
| Quadratic Assignment Problem (QAP), Construct heuristic | paper | |
| Set Cover Problem (SCP), Construct heuristic | paper | |
| Flow Shop Scheduling Problem (FSSP), Construct heuristic | paper | |
| Flow Shop Scheduling Problem (FSSP), Guided local search | paper | |
| Bayesian Optimization, Cost-aware Acquisition Function Design | paper | |
| Co-Bench: Benchmarking Language Model Agents in Algorithm Search for Combinatorial Optimization | paper | |
| Pymoo MOEA/D | paper | |
| Machine Learning | Adversarial Attack, Attack strategy | paper |
| Acrobot, Heuristic (Agent) | ||
| Cart Pole, Heuristic (Agent) | ||
| Mountain Car, Heuristic (Agent) | ||
| Science Discovery | Computational fluid dynamics, Turbulence model design | paper |
| Bacteria Growth, Function | ||
| Oscillator, Equation | ||
| Stress & Strain, Equation | ||
| Math | Admissible Sets | paper |
| coming soon ... |
There are three approaches on LLM interface implementation, check Tutorial on LLM interface implementation for more information.
- Remote LLM API (e.g., GPT4o, GPT3.5, Gemini Pro, Deepseek ...) (<Recommended !!!>)
- Local HuggingFace LLM Deployment (e.g., Llamacode, Llama, Gemma, Deepseek, ...)
- Your Implementation If you want to use your own GPT API or local LLMs deployment, please create and add your interface in LLM
A Step-by-step Tutorial on using LLM4AD to solve your algorithm design task is provided here
Find responses to some common questions in our Discussion Area
The primary project license is BSD-2-Clause; see LICENSE. The bundled DeepMind FunSearch database retains Apache-2.0; see NOTICE and third-party notices. Academic citation and contacting the team for commercial collaboration are voluntary, not license conditions.
If you find LLM4AD helpful please cite:
@article{liu2024llm4ad,
title = {LLM4AD: A Platform for Algorithm Design with Large Language Model},
author = {Fei Liu and Rui Zhang and Zhuoliang Xie and Rui Sun and Kai Li and Xi Lin and Zhenkun Wang and Zhichao Lu and Qingfu Zhang},
year = {2024},
eprint = {2412.17287},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2412.17287},
}This platform is developed and maintained by LLM4AD developer group from the City University of Hong Kong (CityUHK) and the Southern University of Science and Technology (SUSTech). We develop LLM4AD platform for research purposes and hope to contribute to the research area by delivering tools for LLM-based algorithm design methods.
- Contribution: We are more than welcome to contribute (see our 📖 contribution guide) including developing code and ideas to improve our platform.
- Issue: If you find a bug or you have any kind of concern regarding the correctness, please report us an issue.
- Commercial collaboration: Contacting us is welcome and entirely optional; commercial use does not require permission or contact under the license.
If you are interested in LLM4AD or if you encounter any difficulty using the platform, you can:
-
Visit our website LLM4AD Web
-
Visit our collection a collection of resources and research papers on LLM4AD
-
Visit Discussions to connect with other members of our community
-
Join our QQ Group
-
Contact us through email fliu36-c@my.cityu.edu.hk
-
Submit an issue
Any new ideas, features, and improvements are welcomed!
You can contribute to LLM4AD follow our 📖 Contribution Guide.
Thank you for contributing to LLM4AD and welcome to being part of the LLM4AD community! ✨
 Fei Liu |
 Kai Li |
 Rui Sun |
 Julian XIE |
 Shunyu Yao |
 Rui Zhang |
 Weiwei Sun |
 Zhi Zheng |
 Qinglong Hu |