Nemotron cache-aware streaming model fails to generalize to out-of-distribution data after training

[hassan-webm]

I trained nemotron streaming model model for russian on the golos dataset and shows a large WER gap between in-domain validation data and the FLEURS Russian test set, despite never being trained on FLEURS-style data.

Eval Set	WER
In-domain val	7.5%
FLEURS ru test	27%

For reference, a non-streaming Parakeet model finetuned on the same Russian data generalizes well to FLEURS without any read-speech training data, suggesting the generalization gap is at least partially a streaming-specific problem.

Setup:

Base model: nvidia/nemotron-speech-streaming-en-0.6b
Finetuned on: Golos dataset
att_context_size: default, didn't change
Decoding: greedy_batch, cache-aware streaming pipeline

Any help with this will be really appreciated.
@nithinraok @KunalDhawan

[KunalDhawan]

Hi @hassan-webm ,

Thanks for sharing the results. To help debug this further, could you please provide some additional details about your training process?

• Is it possible that the model may have overfit the training set during fine-tuning of nemotron-speech-streaming?
• Did you train both the streaming Nemotron model and the offline Parakeet model for the same number of steps, with similar learning rates, regularization settings, and weight initialization?
• Could you share the training and validation (dev) loss curves, along with the corresponding error rate curves?
• For the FLEURS evaluation, are you observing any particular trend (e.g., increased deletions or substitutions)?

Additionally:

• Could you share the exact config.yaml used for fine-tuning nvidia/nemotron-speech-streaming-en-0.6b?
• The base model is trained with a multi-lookahead approach. Did you follow the same setup during fine-tuning, or did you train with a single lookahead setting (e.g., [70, 13])?
• Could you clarify which chunk size configuration was used for the results you reported?

These details will help us better understand the discrepancy and narrow down the root cause.

Looking forward to your response.

[hamees-sayed]

Hi @KunalDhawan,

Here are the details you asked about:

1. Evaluation Across Epochs

I evaluated at epochs 1, 2, and 10 — all showed the same pattern: low WER on the in-domain set, but high WER on FLEURS.

2. Model Details

• Parakeet model: Initialized from nvidia/parakeet-tdt-0.6b-v2
• Streaming model: Fine-tuned nvidia/nemotron-speech-streaming-en-0.6b on ~60k hours of European language data from espnet/yodas-granary (includes ~11k hours of Russian)

3. Training Loss

Loss curves are attached below.

4. FLEURS Error Analysis

Error Type	Count	% of Errors
Substitutions	282	75.6%
Deletions	47	12.6%
Insertions	44	11.8%

5. Encoder Lookahead Settings

I kept the base model's default lookahead configuration (did not modify it). From hparams.yaml in the training directory:

encoder:
    _target_: nemo.collections.asr.modules.ConformerEncoder
    feat_in: 128
    feat_out: -1
    n_layers: 24
    d_model: 1024
    use_bias: false
    subsampling: dw_striding
    subsampling_factor: 8
    subsampling_conv_channels: 256
    causal_downsampling: true
    reduction: null
    reduction_position: null
    reduction_factor: 1
    ff_expansion_factor: 4
    self_attention_model: rel_pos
    n_heads: 8
    att_context_size:
    - - 70
      - 13
    - - 70
      - 6
    - - 70
      - 1
    - - 70
      - 0
    att_context_style: chunked_limited
    xscaling: false
    untie_biases: true
    pos_emb_max_len: 5000
    conv_kernel_size: 9
    conv_norm_type: layer_norm
    conv_context_size: causal
    dropout: 0.1
    dropout_pre_encoder: 0.1
    dropout_emb: 0.0
    dropout_att: 0.1
    stochastic_depth_drop_prob: 0.0
    stochastic_depth_mode: linear
    stochastic_depth_start_layer: 1

For inference/evaluation, I used the [70, 13] setting.

6. Training Config

name: "NEMOTRON_MULTI_V1_Finetune"

init_from_nemo_model: /data/checkpoints/nemotron-speech-streaming-en-0.6b/nemotron-speech-streaming-en-0.6b_base_eu_multilingual.nemo

model:
  sample_rate: 16000
  compute_eval_loss: false
  use_cer: true
  log_prediction: true

  decoding:
    strategy: greedy_batch
    greedy:
      max_symbols: 10
      use_cuda_graph_decoder: false

  train_ds:
    manifest_filepath: /data/training/eu/train.valid.json
    sample_rate: ${model.sample_rate}
    batch_size: 24
    use_lhotse: false
    shuffle: true
    num_workers: 8
    pin_memory: true
    max_duration: 30
    min_duration: 0.1
    is_tarred: false
    tarred_audio_filepaths: null
    bucketing_strategy: "synced_randomized"
    bucketing_batch_size: null

  validation_ds:
    manifest_filepath: /data/training/eu/val.json
    sample_rate: ${model.sample_rate}
    batch_size: 24
    shuffle: false
    use_lhotse: false
    use_start_end_token: false
    num_workers: 8
    pin_memory: true
    max_duration: 30
    min_duration: 0.1

  test_ds:
    manifest_filepath: /data/training/eu/test.json
    sample_rate: ${model.sample_rate}
    batch_size: 24
    shuffle: false
    use_start_end_token: false
    num_workers: 8
    pin_memory: true
    max_duration: 30
    min_duration: 0.1

  char_labels:
    update_labels: false
    labels: null

  tokenizer:
    update_tokenizer: false
    dir: /data/checkpoints/nemotron-speech-streaming-en-0.6b/merged_tokenizer
    type: bpe

  encoder_config:
    freeze: false

  spec_augment:
    _target_: nemo.collections.asr.modules.SpectrogramAugmentation
    freq_masks: 2
    time_masks: 10
    freq_width: 27
    time_width: 0.05

  optim:
    name: adamw
    lr: 1e-4
    betas: [0.9, 0.98]
    weight_decay: 1e-3
    sched:
      name: CosineAnnealing
      warmup_steps: 5000
      warmup_ratio: null
      min_lr: 5e-6

trainer:
  devices: -1
  num_nodes: 1
  max_epochs: 100
  max_steps: -1
  val_check_interval: 0.25
  accelerator: gpu
  strategy:
    _target_: lightning.pytorch.strategies.DDPStrategy
    gradient_as_bucket_view: true
    timeout:
      _target_: datetime.timedelta
      seconds: 7200
  accumulate_grad_batches: 8  # Effective batch = 8 * 7 GPUs * 24 = 1344
  gradient_clip_val: 1.0
  precision: 32
  log_every_n_steps: 200
  enable_progress_bar: true
  num_sanity_val_steps: 0
  check_val_every_n_epoch: 1
  sync_batchnorm: true
  enable_checkpointing: false
  logger: false
  benchmark: false

exp_manager:
  exp_dir: /data/exps/nemotron_MULTI_v1_finetune
  name: ${name}
  create_tensorboard_logger: true
  create_checkpoint_callback: true
  checkpoint_callback_params:
    monitor: val_wer
    mode: min
    save_top_k: -1
    always_save_nemo: true
    save_last: true
    every_n_epochs: 1
    save_on_train_epoch_end: false
  resume_if_exists: false
  resume_ignore_no_checkpoint: true
  create_wandb_logger: true
  wandb_logger_kwargs:
    name: ${name}
    project: "nemotron_MULTI_finetune"

7. Tokenizer & Vocabulary Expansion

I expanded the tokenizer to cover multilingual tokens using a hybrid update approach in my fine-tuning script. Specifically:

• Extended the BPE tokenizer with new multilingual tokens (vocabulary size increased from original to the merged tokenizer)
• Preserved the original English token weights — the decoder embedding, joint network weights, and biases for the first 1024 tokens (+ the blank token) were copied from the pretrained model and kept intact
• Only the newly added token embeddings were randomly initialized and learned during fine-tuning
• The decoder RNN (dec_rnn) and joint projection layers (joint.pred, joint.enc) were fully restored from the pretrained model

This ensures the model retains its English ASR capability while learning new multilingual tokens on top.