For Canary Varients for a AST task, would it be sufficient to train AST alone, or would ASR need to be trained

[athaze]

I added a new language to Canary's tokenizer. Now I only need to translate this x language to English. Do I need to train the x language on the ASR task first ?

[xXMrNidaXx]

Canary AST vs ASR training — great question!

Short answer: AST-only training CAN work, but joint training is better.

Why:

1. Shared encoder benefits

   • ASR learns acoustic features
   • These transfer to AST
   • Joint training = stronger encoder

2. When AST-only works:

   • Pre-trained encoder already exists
   • Fine-tuning for specific domain
   • Limited compute budget

3. When joint is better:

   • Training from scratch
   • Multiple target languages
   • Maximum accuracy needed

Config for AST-only:

model:
  task: "ast"  # Audio-to-text only
  freeze_encoder: false  # Or true if pre-trained

Config for joint:

model:
  tasks: ["asr", "ast"]
  task_weights:
    asr: 0.5
    ast: 0.5

Recommendation:
If you have an ASR checkpoint, fine-tune for AST. If training fresh, do joint.

We train multilingual models at RevolutionAI. What's your source/target language pair?