Skip to content

[Feature] Add generation_kwargs jinja templating#3684

Open
RawthiL wants to merge 1 commit intoEleutherAI:mainfrom
pnyxai:feature-gen-kwargs-templates
Open

[Feature] Add generation_kwargs jinja templating#3684
RawthiL wants to merge 1 commit intoEleutherAI:mainfrom
pnyxai:feature-gen-kwargs-templates

Conversation

@RawthiL
Copy link
Copy Markdown
Contributor

@RawthiL RawthiL commented Apr 7, 2026

Currently the generation_kwargs of a task definition are fixed once they are defined, meaning that setting something like:

generation_kwargs:
  until: []
  max_gen_toks: 30000
  tools: "[ ... ]"

Would define the generation for all samples in the task with the given parameters. However one could be interested in making this variable, specially for things like tools. This is why we introduce this PR to enable jinja templating of the generation_kwargs fields, using the normal syntax.

This PR enables the construction of generation_kwargs that are modified per sample, like:

generation_kwargs:
  until: []
  max_gen_toks: 30000
  do_sample: false
  tools: "{{tools}}"

During generation, the tools field of the request will be filled with the content of the sample's tools field, changing from sample to sample.

Changes to the codebase are minimum an enable many other applications besides setting tools dynamically.


This PR along with #3685 are part of an effort to enable tool-usage and function calling evaluation on the lm-eval software. We have modified the T-Eval dataset and created a lm-eval compatible one: T-Eval PNYX dataset that includes several tasks for measuring function calls. Tasks are working and we wish to share them.

@RawthiL
Copy link
Copy Markdown
Contributor Author

RawthiL commented Apr 7, 2026

I prefer not to lint at this point since the changes made by the linter will be more than the actual changes introduced in this PR.

Happy to lint before merging tho...

@RawthiL RawthiL marked this pull request as ready for review April 7, 2026 18:03
@RawthiL RawthiL requested a review from 0xSMT as a code owner April 7, 2026 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant