WIP: SALM with NeMo Automodel integration for Nemotron Nano V3 LLM backbone

[pzelasko]

Important

The Update branch button must only be pressed in very rare occassions.
An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Collection: [Note which collection this PR will affect]

Changelog

• Add specific line by line info of high level changes in this PR.

Usage

• You can potentially add a usage example below

# Add a code snippet demonstrating how to use this 

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

• Related to # (issue)

[pzelasko]

Trying to decide if we should make SALM backward compatible with vanilla transformers LLMs (shares lot of logic but gets somewhat complex) or copy this into a new class (cleaner but more duplication). In any case canary-qwen-2.5b released checkpoint must work with the final shape of this PR.

In general, the correct way to fix an unused import in Python is to remove the import statement if the module is never referenced in the file. This reduces visual clutter, avoids implying unnecessary dependencies, and can slightly speed up module import time.

Here, the best fix is to delete the import json line in nemo/collections/common/data/lhotse/text_adapters.py (line 14 in the provided snippet), leaving the rest of the imports unchanged. No additional methods, definitions, or replacement imports are needed, since no code in the shown region uses json. This change preserves all existing functionality because it only removes an unused symbol.

In general, to fix commented-out code you either (a) reinstate it as active code because it is required, or (b) remove it (or convert it into concise explanatory comments) if the behavior is not in use. Here, the function already accepts an enable_thinking flag and the commented block redundantly recalculates it from the content of system/user turns; since this logic is disabled and the docstring describes enable_thinking as a parameter, the least disruptive fix is to remove the commented-out code while preserving the surrounding explanatory comments about step 1. Concretely, in nemo/collections/common/prompts/qwen.py, inside Qwen3PromptFormatter.encode_dialog, delete lines 99–105 that begin with # enable_thinking = True and the subsequent commented for turn in turns: loop. No new methods or imports are needed.

General fix: Ensure that all code paths in configure_gradient_clipping return explicitly and consistently. Since this method is used for side effects, the simplest is to not return the result of super().configure_gradient_clipping(...) and instead always return None (or just return) at the end, after any side-effectful calls.

Concrete best fix for this file:

• In nemo/collections/speechlm2/models/salm.py, in configure_gradient_clipping:
  • Change the if not self._use_fsdp ... branch to call super().configure_gradient_clipping(...) but not return its value.
  • After the conditional logic (and after the if params: block), add an explicit return None (or return) so that every path in the function returns explicitly.
• This keeps behavior identical:
  • In the “no FSDP / no clipping” branch, Lightning’s default configure_gradient_clipping still runs for its side effects.
  • In the FSDP branch, _clip_grad_norm_impl still runs.
  • Callers now always get None (which is what they effectively got before in practice).

No new imports, methods, or definitions are required.

To fix the problem, remove the unused symbol OmegaConf from the import statement so that only DictConfig is imported. This keeps the dependency on omegaconf minimal while preserving all existing functionality, since DictConfig is actively used and OmegaConf is not.

Concretely, in tests/collections/speechlm2/test_salm_automodel_lora.py, at the import block near the top of the file, change the line:

from omegaconf import DictConfig, OmegaConf

to:

from omegaconf import DictConfig

No other code changes, methods, or additional imports are required.

To fix the problem, remove the unused name maybe_install_lora from the multi-name import, while keeping the other imported, used symbols intact. This avoids changing any runtime behavior, because the module will still be imported due to the remaining names, and only the unused symbol binding is removed from this file’s namespace.

Concretely, in tests/collections/speechlm2/test_salm_automodel_lora.py, locate the import block starting at line 30: from nemo.collections.speechlm2.parts.automodel_lora import (...). Edit the parenthesized list to drop the maybe_install_lora entry, leaving LORA_PARAM_PATTERN, ensure_lora_trainable, and make_peft_config unchanged and in place. No other code, imports, or logic need to be modified.

[desh2608]

Trying to decide if we should make SALM backward compatible with vanilla transformers LLMs (shares lot of logic but gets somewhat complex) or copy this into a new class (cleaner but more duplication). In any case canary-qwen-2.5b released checkpoint must work with the final shape of this PR.

(copying my comment from Slack here) In the current PR, does it already work with HF Automodel and NeMo Automodel? If yes, it looks fine to me. Most of the complexity around model loading seems to be in configure_model() and some utility functions in pretrained.py . Other than that, the annoying thing is to have to put DTensor to a full tensor for some operations (I had to do the same for adding audio generation head), but I think it's not too bad.