A meta-optimizer — for optimizing LLM optimizers!
metaTextGrad automatically optimizes LLM-based optimizers like TextGrad and ADAS through meta prompt and structure optimization, achieving stronger performance and better task alignment.
-
[November 2025] The code for metaTextGrad is now released.
-
[September 2025] metaTextGrad is published at NeurIPS 2025. Check out our paper for more details.
-
[May 2025] metaTextGrad is now released. Check out our preprint on arXiv.
To set up the environment for metaTextGrad, please follow the steps below:
conda create -n metatextgrad python=3.10
conda activate metatextgrad
cd textgrad
pip install -e .Since we’ve modified the original TextGrad codebase, make sure to install TextGrad from our repository rather than the official one.
Existing LLM-based optimizers like TextGrad and DSPy are manually designed to perform well across many different tasks. Their prompts are highly general. For example:
You are part of an optimization system that improves text (i.e., variable). You will be asked to creatively and critically improve prompts, solutions to problems, code, or any other text-based variable.
However, this generality can sometimes limit their effectiveness on specific tasks.
When the optimizer only receives noisy feedback, such as overly detailed textual suggestions:
To improve the prompt for the executer and enhance the objective function, consider the following feedback: 1. Explicit Criteria Definition: The prompt should explicitly instruct the executer to define the criteria for optical activity at the beginning of the response. This can prevent ambiguity and ensure that the executer uses the correct scientific principles. For example, the prompt could include a directive to "List the criteria for optical activity before analyzing each compound." 2. Data Verification Directive: Incorporate a step in the prompt that requires the executer to verify the input data against reliable sources. This could be phrased as "Cross-check the properties of each compound with a trusted chemical database before proceeding with the analysis." 3. Structured Logical Reasoning: Encourage a structured approach to reasoning by breaking down the analysis into distinct steps. The prompt could suggest a format like "For each compound, first identify chiral centers, then assess symmetry, and finally determine optical activity". 4. Cross-Referencing Encouragement: ... By incorporating these elements into the prompt, the executer can be guided to produce more accurate and reliable responses, thereby improving alignment with the ground truth answer and enhancing the objective function.
It may incorporate irrelevant information and fail to improve the model effectively.
In contrast, a task-specific optimizer can be explicitly aligned with the target task distribution.
For instance, an optimizer optimizaed for Dyck Languages (bracket matching) might include guidance such as focusing on proper nesting and LIFO order:
You are part of an optimization system specialized in improving prompts for bracket matching and sequence completion tasks. Your role is to enhance prompts that help solve Dyck language problems, which involve proper nesting and closure of different types of brackets ({}, <>, ()). When improving prompts, focus on these critical aspects: (1) maintaining accurate bracket pair matching, (2) preserving the LIFO (Last In First Out) order of nested structures, (3) handling multiple bracket types simultaneously, and (4) ensuring complete closure of all open brackets. You should critically analyze how the prompts can better guide the model to track open brackets, maintain proper nesting order, and systematically complete sequences. Consider incorporating pattern recognition strategies and explicit validation rules in the improved prompts. Your improvements should lead to more reliable and accurate bracket sequence completions.
Such alignment helps the optimizer produce better programs even when the feedback remains noisy. Therefore, rather than designing new optimizers manually for every task, metaTextGrad aims to meta-learn how to adapt optimizers automatically and create optimizers that are both stronger and task-aligned.
Below are examples of how to run metaTextGrad and its baselines.
The baselines include standard prompting methods (e.g., Zero-shot, Few-shot, CoT), as well as optimization-based methods like TextGrad, ADAS-TG, and MIPROv2.
At the end, we show how to run metaTextGrad.
Zero-shot CoT:
python zero_shot_cot.py --task BBH_word_sortingFew-shot CoT with 8 demonstrations:
python few_shot_cot.py --task BBH_word_sortingSelf-consistency (samples multiple reasoning paths and aggregates):
python self_consistency.py --task BBH_word_sortingBest-of-N (selects best answer from multiple generations):
python best_of_n.py --task BBH_word_sortingTextGrad TGD — the gradient-descent-like optimization baseline using textual feedback:
python textgrad_baselines.py --optimizer TGD --task BBH_word_sortingADAS-TG — our implementation of ADAS (Meta Agent Search) within the TextGrad framework:
python textgrad_baselines.py --optimizer PipelineOptimizer --task BBH_word_sortingZero-shot MIPROv2 — no demonstrations:
python dspy_baselines.py --demo 0 --task BBH_word_sortingFew-shot MIPROv2 — with 8 demonstrations:
python dspy_baselines.py --demo 8 --task BBH_word_sortingpython meta_optimization.py --task BBH_word_sortingYou can use the --run_structure_optimization flag to enable meta structure optimization. However, please control the number of optimization epochs and monitor your API usage carefully, as this process may cost tens of dollars or more. Therefore, this option is disabled by default.
You can run other tasks by changing the --task argument.
Available tasks include:
MMLUs(MMLU Abstract Algebra)BBH_word_sortingBBH_dyck_languagesGPQA_diamond
Many existing works greatly inspired this project! Here is a non-exhaustive list:
- 📚 TextGrad — the core codebase our work is built upon.
- 📚 DSPy — a pioneering framework for building LM-based programs in diverse ways.
- 📚 ADAS — uses Meta Agent Search to show that agents can invent novel and powerful designs; we use this idea within the TextGrad framework to implement ADAS-TG.
@inproceedings{
xu2025metatextgrad,
title={metaTextGrad: Automatically optimizing language model optimizers},
author={Guowei Xu and Mert Yuksekgonul and Carlos Guestrin and James Zou},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
year={2025},
url={https://openreview.net/forum?id=10s01YrlKp}
}We are grateful for all the help we got from our contributors!
Guowei Xu |
Mert Yuksekgonul |