Skip to content

Add --config_source pretrained to capture full model config via model_id - #181

Draft
kiszk wants to merge 3 commits into
mainfrom
feat/config-source-from-pretrained
Draft

Add --config_source pretrained to capture full model config via model_id#181
kiszk wants to merge 3 commits into
mainfrom
feat/config-source-from-pretrained

Conversation

@kiszk

@kiszk kiszk commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a --config_source option to auto_generate_module_config.py that
controls how a module's PretrainedConfig constructor argument is recorded in
the generated YAML.

  • kwargs (default, unchanged): rebuild the config from a handful of extracted
    dimensions (config_path + config_kwargs). Library defaults fill in
    everything else — the historical behaviour.
  • pretrained (new): record only the model_id so the OOT test framework can
    load the full, faithful config with AutoConfig.from_pretrained(model_id).

Motivation

The kwargs path extracts only ~6 fields (hidden_size,
num_attention_heads, num_key_value_heads, intermediate_size,
max_position_embeddings, _attn_implementation). Every other field
(rope_theta, rope_scaling, attention_bias, head_dim, logits_scaling,
...) is dropped to library defaults, so a module rebuilt from the YAML can
diverge from the real model. Recording the model_id and reloading the config
at test time preserves every field the model actually had.

There are two other motivations for this PR (out of scope of this PR)

  • Will support calling LLM() for instead of instantiating config_cls to support this architecture
  • Will introduce opportunities to apply something (e.g. monkey patches and optimizations of device layouts of weight tensors beyond generic layout) at a model loading time.

Changes (utils/module_discovery/auto_generate_module_config.py)

  • New CONFIG_SOURCE_KWARGS / CONFIG_SOURCE_PRETRAINED constants and a shared
    _build_config_arg_spec(config, config_source, model_id) helper that produces
    the config-arg spec, replacing the two inlined spec-building blocks in
    capture_constructor_info.
  • In pretrained mode the spec emits model_id plus a small config_overrides
    carrying the resolved _attn_implementation (which from_pretrained may
    leave as None), keeping the standalone module on the same attention path
    used at capture time. config_path is still recorded as a human-readable
    hint.
  • ModuleInfoCapture.__init__ gains config_source / model_id (validated),
    so switching works both from the CLI and from the driver API used by
    generate_*_config.py scripts.
  • _convert_constructor_arg_to_sample_input emits the model_id /
    config_overrides form when present, else the existing config_kwargs form.
  • New --config_source {kwargs,pretrained} CLI flag (default kwargs); main
    passes it plus model_id=args.model_path into ModuleInfoCapture.

Fully backward compatible: the default (kwargs) reproduces the previous output
byte-for-byte, and existing generated YAMLs are unaffected.

Related

This is the generator half of the change. The paired consumer change (the OOT
framework resolving model_id via AutoConfig.from_pretrained) is submitted
separately against the torch-spyre repo.

#173

Testing

  • Verified the spec logic in isolation: kwargs and pretrained produce the
    expected shapes, pretrained without a model_id raises, and
    ModuleInfoCapture rejects an invalid config_source.
  • Usage:
    python utils/module_discovery/auto_generate_module_config.py --model_path ibm-granite/granite-3.3-8b-instruct --config_source pretrained -o /tmp/granite_pretrained.yaml
  • Verified end-to-end run through the OOT runner on the Spyre

kiszk added 2 commits July 15, 2026 07:05
Signed-off-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>
Signed-off-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>
@kiszk
kiszk force-pushed the feat/config-source-from-pretrained branch from f99ca9a to b43aa98 Compare July 15, 2026 07:46
Signed-off-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant