Manual decoding_input_ids/prefix for the decoder injection

[azziko]

Is your feature request related to a problem? Please describe.
Real time inference using offline models with streaming policies often requires directly working with history for more context aware inference. Hence the need for manual decoder_input_ids injection. For example, streamAtt, alignAtt. Also see hlt-mt/simulstream#17 (comment)

Primarily, I would like this feature for Canary-v2 model.

Describe the solution you'd like

I would see this feature best abstracted in a top level API call, such as transcribe. For example, the following snippet is from the SeamlessM4t model: https://github.com/hlt-mt/simulstream/blob/82c75ee77ee39a7f672cef12f75373035f94c73a/simulstream/server/speech_processors/seamless_streamatt.py#L175

For the Canary-v2 purposes, I see one of the solutions to add decoder_input_ids as a field in MultitaskTrasncriptionConfig and then in the _transcribe_forward implementation of the EncDecMultiTask model to replace the decoder_input_ids provided from the config, if any.

Describe alternatives you've considered

Any clean and easy way to do inference and have control over decoder_input_ids without going into the lower level API would suffice. I would like to hear your thought on it. Maybe injection via the prompt could work too.

Additional context

If you find the solution described above as acceptable(and the feature in general), I would be happy to PR the implementation. Thank you

[azziko]

I found a way to inject context via

NeMo/nemo/collections/common/prompts/canary2.py

Line 55 in 0ae9f88

"decodercontext": Modality.Text,

[azziko]

Reopening the request because it was not quite what I needed. This thread explains it better: openai/whisper#117 (comment)

Would it be a good idea to extend

NeMo/nemo/collections/common/prompts/canary2.py

Line 52 in d7c366c

"template": f"{CANARY2_BOCTX}|decodercontext|{CANARY_BOS}|emotion||source_lang||target_lang||pnc||itn||timestamp||diarize|",

with optional:

"template": f"{CANARY2_BOCTX}|decodercontext|{CANARY_BOS}|emotion||source_lang||target_lang||pnc||itn||timestamp||diarize|prefix_ids|",

@nithinraok @andrusenkoau

[pzelasko]

I think what you actually want is another "role" in the prompt formatter, sth like "user_prefix" with template "|prefix_text|", and then specify "user" role for first chunk and "user_prefix" role for the next chunks. Is that right?
I don't know whether the pretrained checkpoints would handle that well, but if you want to experiment with that and get a good outcome, we can consider accepting a PR.

[azziko]

@pzelasko hello,

That’s a good idea, thank you. I will try it

[azziko]

@pzelasko works great for me, I created a small PR. I tested it on multiple inputs and the model was indeed behaving as expected. If you could suggest a framework to write the tests for the feature, I would happily do so. I was going to write UTs for the prompt formatter, but I didn't find any tests for Canary2, so I left it as is.

If you could review, I would be really greatuful.