feat: rebase fork onto upstream HEAD with all custom Corti commits - #3

Signed-off-by: aandrusenko <aandrusenko@nvidia.com>

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

Signed-off-by: L0RD-ZER0 <68327382+L0RD-ZER0@users.noreply.github.com>

…15610)

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

…IA-NeMo#15612)

- add /debug-training-logs slash command that analyzes SLURM worker
  stderr logs and optional AIStore daemon logs to find root causes
  of distributed training failures
- covers NCCL timeout analysis: distinguishes straggler ranks
  (stuck in data loading) from GPU fabric hangs by comparing
  enqueued vs completed work across ALL ranks
- includes AIStore log parsing: file time ranges, timezone
  verification, error counter tracking, proxy/target correlation
- documents NeMo-specific sync points (PreemptionCallback broadcast,
  checkpoint broadcasts, DDP allreduce) that can cause rank desync
- documents Lhotse data loading pitfalls: missing read timeouts,
  m4a BytesIO extension loss, idle connection resets, fault_tolerant
  silent drops
- includes instructions to obtain logs via scp and download AIS
  daemon logs via ais CLI with env var auth

Signed-off-by: Abhishek Gaikwad <gaikwadabhishek1997@gmail.com>

Signed-off-by: Ryan <rlangman@nvidia.com>

…ducibility, speed up training and support half precision inference (NVIDIA-NeMo#15583)

* Add new formatter and force codec to runs in the same device of TTS model

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Set find_unused_parameters=False as default to speedup training

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Apply isort and black reformatting

Signed-off-by: Edresson <Edresson@users.noreply.github.com>

* Fix model input data type

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Apply isort and black reformatting

Signed-off-by: Edresson <Edresson@users.noreply.github.com>

* Add support for half precision inference

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Apply isort and black reformatting

Signed-off-by: Edresson <Edresson@users.noreply.github.com>

* Fix RMSnorm

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Update debug_dtype default parameter

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Apply isort and black reformatting

Signed-off-by: Edresson <Edresson@users.noreply.github.com>

* Add unit tests for duplex formatters

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Apply isort and black reformatting

Signed-off-by: Edresson <Edresson@users.noreply.github.com>

* Replace fastcopy with deepcopy

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Apply isort and black reformatting

Signed-off-by: Edresson <Edresson@users.noreply.github.com>

* Remove unecessary comments

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Update unit tests

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Add function for dtype debug and create a method to check codec precision

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Apply isort and black reformatting

Signed-off-by: Edresson <Edresson@users.noreply.github.com>

---------

Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson <Edresson@users.noreply.github.com>
Co-authored-by: Edresson <Edresson@users.noreply.github.com>

* cleunup tail padding algorithms

Signed-off-by: naymaraq <dkaramyan@nvidia.com>

* remove tail margin algorithms at all

Signed-off-by: naymaraq <dkaramyan@nvidia.com>

* Apply isort and black reformatting

Signed-off-by: naymaraq <naymaraq@users.noreply.github.com>

---------

Signed-off-by: naymaraq <dkaramyan@nvidia.com>
Co-authored-by: naymaraq <dkaramyan@nvidia.com>

NVIDIA-NeMo#15447)

* WIP: bringing Yifan's changes to main

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* Add workaround for exp_manager issue

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* Support reading indexed JSONL datasets with ShareGPT format

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Support reading indexed tarred datasets with ShareGPT format

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Refactor for compactness

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fixes for real-life data

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fixes for real-life data

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fixes for real-life data

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fixes for missing wids-meta.json

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fixes for tarfile edge cases

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fixes for real-world tar files

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* move salm llm init to configure_model

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* fix: delayed perception init

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* Add AutomodelParallelStrategy for Automodel LLM support

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Replace HF Automodel with NeMo Automodel for SALM's LLM backbone

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Update salm default config with new options

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Init fixes

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fix dtype initialization

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fix mesh selection for speech encoder

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fix for mismatched device_mesh axis names in gradient clipping - use automodel's utility

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fix for using embed_tokens in FSDP context before running forward on full LLM

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Definitive fix for using embed_tokens outside of llm with fsdp

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* this version actually works with Automodel

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* fix from_pretrained with transformers v5

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* fix from_pretrained with transformers v5

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* fix generate/eval

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* fix to_hf

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fixes for AutoTokenizer decoding in v5

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Flag to run configure_model() at the end of __init__ for safetensors converted models

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* preliminary: support distributed models in to_hf.py

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* fix passing automodel kwargs

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* fix

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Enable inference with model parallelism

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fix for lightning save_hyperparameters() call

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fix for loading into DTensor

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Accelerate loading DTensor

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Accelerate loading DTensor

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Accelerate loading DTensor

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fix for pe buffers not in ckpt (essentially strict=False)

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Add Nemotron Nano v3 prompt formatter with <think> reasoning support

Implements NemotronNanoV3PromptFormatter (NAME="nemotron-nano-v3") using
ChatML-style <|im_start|>/<|im_end|> template with encode_dialog override
that handles: auto-insert empty system turn, history thinking truncation,
<think></think> prepend for non-thinking assistant turns, and dynamic
inference prefix (thinking on/off). Includes Lhotse Cut integration via
registered_prompt_format_fn. Verified against HF apply_chat_template for
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 (both string and token match).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Automodel LoRA support

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fixes for model parallel

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* LoRA fix

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* small ckpt conversion/inference fix

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Separate SALM and SALMAutomodel into independent classes

Restore salm.py to its original HF Transformers + PEFT LoRA implementation
from main, and extract the NeMo Automodel-based implementation into a new
SALMAutomodel class in salm_automodel.py. This keeps both backends available
and independent, with scripts auto-detecting the model class from config.json.

- salm.py: restored from main (eager init, HF PEFT, move_embedding)
- salm_automodel.py: new file with SALMAutomodel (deferred init, automodel LoRA)
- salm_train.py: selects model class via model.use_nemo_automodel config key
- salm_eval.py/salm_generate.py: auto-detect model class from config.json
- salm_automodel.yaml: new config for SALMAutomodel training
- Tests split into test_salm.py (CPU) and test_salm_automodel.py (CUDA)
- New functional test SPEECHLM_Automodel_Training_SALM.sh

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* fix linters

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Add SALMAutomodel docs and speechlm2 pip extra

Add documentation for SALMAutomodel (NeMo Automodel variant of SALM)
across all speechlm2 doc pages: intro, models, configs, and
training_and_scaling. Create pip install nemo-toolkit[speechlm2] extra
that composes speechlm2-only (nemo_automodel git dep) + asr + tts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Add SALMAutomodel tutorial notebook and fix EP/FSDP2 docs

Add tutorials/speechlm2/SpeechLM_With_NeMo_Automodel.ipynb covering
the full pipeline: data download, training, checkpoint conversion, and
evaluation with Nemotron Nano V3 MoE backbone on 2 GPUs.

Fix docs to clarify that Expert Parallelism reuses the FSDP2
data-parallel axis — dense layers are sharded via FSDP2 while MoE
layers use EP on the same GPUs, not a separate dimension.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Fix uv torch index conflict for speechlm2 extra

The docs CI runs `uv sync --all-extras --all-groups` which resolves
the speechlm2 extra pulling nemo_automodel from git. uv treats git
source deps as workspace members and applies their [tool.uv.sources],
causing a conflict: Automodel maps torch to per-platform indexes
while NeMo defaulted to PyPI for all platforms.

Add matching [tool.uv.sources] for torch to pyproject.toml and
regenerate uv.lock with nemo_automodel included.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Remove direction arg

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* Apply isort and black reformatting

Signed-off-by: pzelasko <pzelasko@users.noreply.github.com>

* fix linter

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* fix tests

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* fixes

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* fixes for trust_remote_code

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* Apply isort and black reformatting

Signed-off-by: pzelasko <pzelasko@users.noreply.github.com>

* Add explicit enable_thinking support to SALM eval paths

* Apply isort and black reformatting

Signed-off-by: pzelasko <pzelasko@users.noreply.github.com>

* fix inference with ep_size=1 for automodel models

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* Fixes

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* Fixes for inference and tutorial

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* Apply isort and black reformatting

Signed-off-by: pzelasko <pzelasko@users.noreply.github.com>

* Remove deprecated activation_checkpointing parameter everywhere

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fix CI

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* Fix to_hf.py crash when run without torchrun

Guard dist.init_process_group on RANK env var presence so the script
works with plain `python` (single-file checkpoints) as well as
`torchrun` (distributed checkpoints).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Apply suggestions from code review

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* Add flashoptim support and bf16-automodel half precision setup

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* Apply isort and black reformatting

Signed-off-by: pzelasko <pzelasko@users.noreply.github.com>

* patch flashoptim handling of unevenly sharded state dicts

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* Reproducibility fix

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* Refactor AutomodelPrecision to FlashPrecision to enable re-use by other collections in subsequent PRs

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* Apply isort and black reformatting

Signed-off-by: pzelasko <pzelasko@users.noreply.github.com>

* Address code review

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* disable linter

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fix test

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* fix for torch.compile config

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* Apply isort and black reformatting

Signed-off-by: pzelasko <pzelasko@users.noreply.github.com>

* fix tests

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Dataloader DP rank patch for Automodel's device_mesh

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* Apply isort and black reformatting

Signed-off-by: pzelasko <pzelasko@users.noreply.github.com>

* fix sloppy fix

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* fix CI HF tokenizer download issue

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Add tests for correct DP rank resolution in the dataloader

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* Apply isort and black reformatting

Signed-off-by: pzelasko <pzelasko@users.noreply.github.com>

* xfail tests with corrupted tokenizer in CI

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Update test pytorch version safeguard

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fix new peft version requiring newer torchao than available in CI container

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fixes

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

* Bump Automodel pin for transformers compat

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

---------

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>
Signed-off-by: Piotr Żelasko <petezor@gmail.com>
Signed-off-by: pzelasko <pzelasko@users.noreply.github.com>
Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: pzelasko <pzelasko@users.noreply.github.com>
Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

…sor (NVIDIA-NeMo#15613)

* Add CLAUDE.md and symlinks for Codex/Cursor

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Address review + babysit PR action + 2 skills (babysit-pr, fix-issue)

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fix double-posting of 'taking over' comment and change the text

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fix truncated sentence

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Make babysitter less autonomous

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Harden babysitter: speech_team-only, fork guards, drop reaction path

Addresses Charlie's review on NVIDIA-NeMo#15613. Adds per-job preflights that verify
the acting user (label sender / approval commenter / execute-fix sender) is
an active NVIDIA-NeMo/speech_team member, and fork-guards every PR-scoped
job so the babysitter never runs on forks. Also blocks manual-label bypass
by requiring a bot-authored plan comment before execute-fix proceeds, and
drops the cron+reaction approval path in favor of natural-language replies.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Signed-off-by: Piotr Żelasko <petezor@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…VIDIA-NeMo#15619)

Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.5.3 to 7.5.5.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md)
- [Commits](protobufjs/protobuf.js@protobufjs-v7.5.3...protobufjs-v7.5.5)

---
updated-dependencies:
- dependency-name: protobufjs
  dependency-version: 7.5.5
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [mako](https://github.com/sqlalchemy/mako) from 1.3.10 to 1.3.11.
- [Release notes](https://github.com/sqlalchemy/mako/releases)
- [Changelog](https://github.com/sqlalchemy/mako/blob/main/CHANGES)
- [Commits](https://github.com/sqlalchemy/mako/commits)

---
updated-dependencies:
- dependency-name: mako
  dependency-version: 1.3.11
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [pillow](https://github.com/python-pillow/Pillow) from 12.1.1 to 12.2.0.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](python-pillow/Pillow@12.1.1...12.2.0)

---
updated-dependencies:
- dependency-name: pillow
  dependency-version: 12.2.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…-NeMo#15586)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 6.3.6 to 6.4.2.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v6.4.2/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v6.4.2/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-version: 6.4.2
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@users.noreply.github.com>
Co-authored-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@users.noreply.github.com>

Bumps [pygments](https://github.com/pygments/pygments) from 2.19.2 to 2.20.0.
- [Release notes](https://github.com/pygments/pygments/releases)
- [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES)
- [Commits](pygments/pygments@2.19.2...2.20.0)

---
updated-dependencies:
- dependency-name: pygments
  dependency-version: 2.20.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…VIDIA-NeMo#15554)

Bumps [picomatch](https://github.com/micromatch/picomatch) from 4.0.2 to 4.0.4.
- [Release notes](https://github.com/micromatch/picomatch/releases)
- [Changelog](https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md)
- [Commits](micromatch/picomatch@4.0.2...4.0.4)

---
updated-dependencies:
- dependency-name: picomatch
  dependency-version: 4.0.4
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…A-NeMo#15560)

Made-with: Cursor

Signed-off-by: stanley1208 <stanley.mei08@gmail.com>

…-NeMo#15627)

The PR Babysitter never investigated real CI failures. Check-runs
produced by GHA jobs authenticated with the default GITHUB_TOKEN do
not fire `check_run` events on downstream workflows (GitHub's
recursion guard). As a result, `check-label-for-ci` skipped on every
CI failure it was meant to handle — verified on PR NVIDIA-NeMo#15626 where all
31 recent check_run-triggered runs skipped every job despite multiple
failing checks (Nemo_CICD_Test, Nemo_Linting_Test, etc.).

Replace the `check_run: [completed]` trigger with an explicit
`workflow_run: [completed]` list covering the CI workflows that can
fail on a PR (CICD NeMo, PyLint/flake8, wheel build, __init__ check,
copyright, CI-Install-Check, CodeQL, secrets detector). Intentionally
omitted: "Isort and Black Formatting" (auto-pushes fixes), the
babysitter itself, and labeler/relabel bots.

Update `check-label-for-ci`, the investigate prompt, and
`ping-author-on-failure` to read `github.event.workflow_run.*`
instead of `github.event.check_run.*` (conclusion, pull_requests,
head_sha, name). Drop the `reformat_with_isort_and_black` name filter
— filtering is now done at the trigger level by not listing that
workflow.

Semantics change: one investigation per failing workflow (not per
failing check). The existing prompt already instructs Claude to look
at all failing checks on the PR, so coverage is unchanged and the
noise is lower.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…o#15628)

The `r**` glob matched any branch starting with `r` (e.g.
`remove-…`), causing feature branches to trigger the Test PyPI
upload and collide on the generated version.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

…NeMo#15626)

* Remove nemo_log_globalrank-N_localrank-M.txt file creation

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* revert one change

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* fix linter

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Addres PR review, remove 2 flags that are no longer used

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

---------

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

…pping (NVIDIA-NeMo#15561)

Made-with: Cursor

Signed-off-by: stanley1208 <stanley.mei08@gmail.com>

…ith backbone-native chat_template (NVIDIA-NeMo#15623)

* to_hf.py: unconditionally produce vLLM-ready HF checkpoints

`to_hf.py` now always emits a checkpoint that can be served by vLLM's
SpeechLM plugin: architecture/model_type fields in config.json, the
backbone's canonical chat_template in tokenizer_config.json, audio
placeholder registered on the tokenizer, and a minimal generation
config. The previous `vllm: bool` opt-in flag is gone -- HF and vLLM
loaders now share the same on-disk artifact (pzelasko's ask on the
closed PR NVIDIA-NeMo#15617 thread).

Key bits:
- `_detect_vllm_architecture` inspects the backbone's HF config to
  pick the right vLLM plugin class. Fail-fast ValueError on missing
  `architectures` rather than silently defaulting to 'Std' (also
  addresses pzelasko's review comment about the broad except).
- `prepare_for_vllm` is invoked unconditionally after save, wrapped in
  `_try_prepare_for_vllm` which downgrades a `ValueError` to a warning
  so legacy callers that never needed vLLM (e.g., NeMo SALM loading
  the same dir) still get a clean HF-only checkpoint.
- Tokenizer is re-saved from the backbone (brings its native
  chat_template along) + augmented with `<|audio|>`; extra_special_tokens
  is normalized to a dict so vLLM's AutoTokenizer can load it.
- For reasoning backbones (nemotron-nano-v3), the exported
  chat_template's `enable_thinking` default is flipped to False so
  vLLM's request-time render matches training-time render; otherwise
  vLLM silently prepends `<think>\n` to every assistant turn and WER
  regresses. Verified librispeech-pc WER 1.57 (== baseline) after this
  fix; without it WER regressed to 5.92.

hf_hub.py: setdefault `model_type` and `architectures` in
`HFHubMixin.save_pretrained` so NeMo-saved SpeechLM checkpoints carry
the metadata vLLM / transformers need to identify them.

Made-with: Cursor
Signed-off-by: Dongji Gao <dongjig@nvidia.com>

* to_hf.py: add missing return-type hints per NeMo PR checklist

Five top-level helpers (load_checkpoint, setup_distributed_from_config,
consolidate_state_dict, save_hf_checkpoint, main) lacked return types.
Uses Any for setup_distributed_from_config's AutomodelParallelStrategy
return to avoid adding an import just for typing; concrete types
everywhere else. Same fix previously applied on pr/vllm-plugin.

Made-with: Cursor
Signed-off-by: Dongji Gao <dongjig@nvidia.com>

* Apply isort and black reformatting

Signed-off-by: Dongji Gao <dongjig@nvidia.com>

* to_hf.py: rescue chat_template.jinja before deleting it

Modern HuggingFace transformers (~4.42+) moves long chat_template
strings out of tokenizer_config.json into a separate
chat_template.jinja file to keep the JSON readable. Qwen3-1.7B's
4168-char template triggers this split; Nemotron-Nano's shorter
template stays inline.

The old code deleted chat_template.jinja before reading
tokenizer_config.json, assuming the inline copy was always complete.
For Qwen3 that meant the exported checkpoint shipped with an empty
chat_template -- vLLM's apply_chat_template returned a prompt
without the <|audio|> placeholder, which broke multimodal prompt
replacement (Failed to apply prompt replacement for mm_items['audio'][0]).

Now read chat_template.jinja, inline it into tokenizer_config.json
when non-empty, and only then delete the file. Nemotron's inline-only
path is unchanged because .jinja doesn't get written for small
templates.

Made-with: Cursor
Signed-off-by: Dongji Gao <dongjig@nvidia.com>

* to_hf.py: force tokenizer_class=PreTrainedTokenizerFast for vLLM compat

Newer NeMo containers (e.g. nemo-25.11-pytorch2.9-automodel-03apr26)
wrap AutoTokenizer.from_pretrained(trust_remote_code=True) in a
NeMo-internal TokenizersBackend class. save_pretrained then writes
'tokenizer_class: TokenizersBackend' to tokenizer_config.json --
not in HF transformers' registry, so vLLM's
AutoTokenizer.from_pretrained crashes at server load:

  ValueError: Tokenizer class TokenizersBackend does not exist or
  is not currently imported.

The underlying tokenizer.json is a valid HF fast tokenizer regardless
of which wrapper produced it; force the class name back to
PreTrainedTokenizerFast so downstream HF-based loaders (including
vLLM) can round-trip the config.

Made-with: Cursor
Signed-off-by: Dongji Gao <dongjig@nvidia.com>

* QwenPromptFormatter: align with Qwen3 enable_thinking=False pre-training

Qwen3's chat_template injects '<think>\n\n</think>\n\n' before assistant
content when enable_thinking=False (the 'no reasoning' mode, which is
what SpeechLM ASR wants). The old QwenPromptFormatter didn't include
this prefix, so SpeechLM fine-tunes trained through it showed the model
a turn shape that's different from Qwen3's pre-training convention.

Bake NO_THINK_PREFIX into both INFERENCE_PREFIX and (transitively via
INFERENCE_PREFIX) the assistant template, so future fine-tunes produce
training data byte-identical to 'apply_chat_template(enable_thinking=
False)'. Existing checkpoints are unaffected -- the change only kicks
in the next time you retrain with prompt_format=qwen.

Test: updated hardcoded expected strings to match Qwen3 jinja output
for single-turn training and inference.

Made-with: Cursor
Signed-off-by: Dongji Gao <dongjig@nvidia.com>

* to_hf.py: read audio_locator_tag from model config as SoT

Removes the hardcoded _AUDIO_TOKEN constant and reads the audio placeholder
from model_cfg["audio_locator_tag"], raising ValueError if missing. This
ensures the exported config.json, added tokenizer symbols, and
extra_special_tokens dict all reference the same source of truth as training,
avoiding silent drift between train-time and inference-time audio tokens.

Signed-off-by: Dongji Gao <dongjig@nvidia.com>
Made-with: Cursor

* NemotronNanoV3PromptFormatter: fix past-asst-no-think + test coverage

Before this fix, the formatter only normalized past assistant turns that
already contained <think>...</think> tags, so a past assistant turn without
any think tags would emit as "<|im_start|>assistant\nTEST<|im_end|>\n" while
the HF jinja template emits "<|im_start|>assistant\n<think></think>TEST<|im_end|>\n"
(jinja unconditionally injects an empty think block for content lacking both
tags). This caused a silent train/inference-template divergence for
multi-turn dialogs without reasoning history.

Fix step 3 in encode_dialog to handle all three cases symmetrically:
  - both tags present -> truncate to "<think></think>" + post-</think> content
  - neither tag present -> prepend "<think></think>"
  - only one tag present -> leave as-is (matches jinja)

Also adds three tests to fill previously-missing coverage:
  - training multi-turn with past assistant missing think tags
    (regression test for the fix above)
  - inference multi-turn with enable_thinking=False
  - inference multi-turn with enable_thinking=True

All 12 non-HF tests pass in the NeMo 25.11 container.

Signed-off-by: Dongji Gao <dongjig@nvidia.com>
Made-with: Cursor

* to_hf.py: stop patching chat_template's enable_thinking default

The previous in-place string-replace flipped Nemotron's
``enable_thinking`` default from True to False so that default vLLM
inference (with no ``chat_template_kwargs``) would match SpeechLM
training rendering. This approach is fragile (silently no-ops if
upstream changes the template) and surprising for downstream consumers
of the exported checkpoint.

Serving callers should instead pass ``chat_template_kwargs={"enable_thinking": False}``
(or the OpenAI-API equivalent) at inference time to opt out of thinking.
This keeps the exported chat_template byte-identical to the backbone's
canonical HF template.

Signed-off-by: Dongji Gao <dongjig@nvidia.com>
Made-with: Cursor

* tests: add unit tests for to_hf.py prepare_for_vllm

Covers the behavior introduced / changed in this PR:
  * Error paths for missing pretrained_llm and missing audio_locator_tag
  * config.json patching (model_type, architectures, audio_locator_tag SoT)
  * Audio token registration (add_special_tokens called only when missing
    from the backbone vocab)
  * tokenizer_config.json normalization (dict-form extra_special_tokens,
    forced tokenizer_class=PreTrainedTokenizerFast)
  * chat_template.jinja rescue (inlined back into tokenizer_config.json
    and the separate .jinja file removed)
  * chat_template is byte-identical after prep (regression guard for the
    removal of the enable_thinking default-flip)
  * generation_config.json carries the tokenizer's eos_token_id

The script lives under examples/ and is loaded via importlib; AutoTokenizer
and _detect_vllm_architecture are patched so the tests run fully offline.
9 tests pass in 0.35s in the NeMo 25.11 container.

Signed-off-by: Dongji Gao <dongjig@nvidia.com>
Made-with: Cursor

* Apply isort and black reformatting

Signed-off-by: Dongji Gao <dongjig@nvidia.com>

* Revert "QwenPromptFormatter: align with Qwen3 enable_thinking=False pre-training"

This reverts commit 0245881.

Piotr raised a concern that baking ``NO_THINK_PREFIX`` into
``QwenPromptFormatter`` changes the turn shape seen by in-flight fine-tunes
like canary-qwen-2.5b, which were trained on the prior (no-think-prefix)
formatter output. Re-rendering the same data through the updated formatter
would silently shift the prompt distribution and break those checkpoints.

Reverting the bake keeps the ``qwen`` prompt format byte-identical to the
version those checkpoints saw during training. Any future fine-tune that
actually wants the ``<think></think>`` empty-reasoning prefix should use
``Qwen3PromptFormatter`` (already handles it via ``enable_thinking=False``)
or explicitly include the prefix in training data, rather than flipping the
default for all ``qwen`` consumers.

Signed-off-by: Dongji Gao <dongjig@nvidia.com>
Made-with: Cursor

---------

Signed-off-by: Dongji Gao <dongjig@nvidia.com>

Cherry-pick of upstream NeMo PR NVIDIA-NeMo#15411 (NVIDIA-NeMo#15411).

Adds token_durations tracking to BatchedBeamHyps for TDT models, fixes
timestamp semantics (store start frame, not start+duration), and downgrades
preserve_alignments NotImplementedError to a warning in the malsd_batch and
maes_batch computers so that compute_timestamps=True works with beam search.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>