A dub that tells you what to check
Every step of automatic dubbing is a model, and errors compound: a misheard word becomes a confident wrong translation, then fluent synthetic speech that says something wrong. DUBBE's answer is a second output next to the dub: a short list of sentences a person should check, shown on the video's timeline.
A capstone project
Applied AI & Business Capstone Design, Inha University × HumbleBeeAI, Fall 2026: Team 1, project C2b, Video Translation and Dubbing.
Team
Models and licences
Every step is an open model that we run ourselves, on rented GPUs. The live Korean and Uzbek voices (MMS) are the part that is still non-commercial.
Speech recognition
Whisper large-v3 / large-v3-turbo (faster-whisper); Uzbek: rubaistt v2
MIT · Apache 2.0Voice and background
Demucs (htdemucs)
MITWho speaks when
NVIDIA Nemotron 3 Diarization, with TitaNet-large voice fingerprints
OpenMDW 1.1 · CC BY 4.0Translation
MADLAD-400 3B (int8, CTranslate2), fine-tuned by the team for Uzbek; Korean ↔ English second opinion: Qwen3-8B
Apache-2.0 · data ODC-BYEnglish and Korean voices
Ming-3.0-tts 0.5B and CosyVoice 3, from Kokoro-82M preset recordings
MIT · Apache 2.0Uzbek and Kazakh voices
Navoiy TTS (CosyVoice 2); Piper with the ISSAI Kazakh voice
Apache 2.0 · MIT, voice CC BY 4.0The other languages
Chatterbox Multilingual; MMS as a fallback
MIT · MMS CC BY-NC 4.0Emotion and sounds heard
emotion2vec+ large (FunASR); AST on AudioSet
FunASR model licence · BSD 3-ClauseEvery model, its licence and the exact version we run (29)
| Model | Step | Licence | Commercial use | Version |
|---|---|---|---|---|
| Systran/faster-whisper-large-v3 | Speech recognition | MIT (conversion of openai/whisper-large-v3, Apache-2.0) | yes | edaa852ec7 |
| mobiuslabsgmbh/faster-whisper-large-v3-turbo | Speech recognition, live dubbing, heard-back check | MIT (conversion of openai/whisper-large-v3-turbo, MIT) | yes | 0a363e9161 |
| islomov/rubaistt_v2_medium | Speech recognition (Uzbek) | Apache-2.0 | yes | af46011761 |
| htdemucs | Voice and background separation | MIT (Demucs v4 package and weights, dl.fbaipublicfiles.com) | yes | 955717e8-8 |
| nvidia/Nemotron-3-Diarization | Who speaks when | OpenMDW-1.1 | yes | f667ed73ae |
| nvidia/speakerverification_en_titanet_large | Voice fingerprints (speakers) | CC BY 4.0 TitaNet-large by NVIDIA, CC BY 4.0 |
yes | 0dc382f401 |
| microsoft/wavlm-base-plus-sv | Voice fingerprints (fallback without the NeMo environment) | none on the card (upstream microsoft/unilm repository is MIT); team rule 2026-11-11, no licence = MIT | yes | feb593a6c2 |
| Nextcloud-AI/madlad400-3b-mt-ct2-int8 | Translation (MADLAD-400 3B, int8 CTranslate2 conversion of google/madlad400-3b-mt) | Apache-2.0 | yes | aa32bbdeba |
| dubbe-ai/madlad400-3b-uz-ct2-int8 | Translation, MADLAD-400 3B fine-tuned by the team for Uzbek (C81; LoRA merged into google/madlad400-3b-mt, int8 CTranslate2). Local folder, loaded when DUBBE_MADLAD points at it; not uploaded | Apache-2.0 (derivative of google/madlad400-3b-mt); training data OPUS NLLB v1, ODC-BY 1.0 Fine-tuned on OPUS NLLB v1 (data mined by Meta AI, released by AllenAI; ODC-BY 1.0; opus.nlpl.eu) |
yes | sha256-427 |
| Qwen/Qwen3-ASR-1.7B-hf | Second listen (a second recogniser next to Whisper; flags sentences the two hear differently) | Apache-2.0 | yes | bcd2b5b7f3 |
| google/metricx-24-hybrid-large-v2p6-bfloat16 | Translation quality without a reference (MetricX-24, QE mode) | Apache-2.0 | yes | febb720e29 |
| google/mt5-xl | Tokenizer of MetricX-24 (only spiece.model and its configs are downloaded) | Apache-2.0 | yes | 63fc6450d8 |
| qwen3:8b | Second opinion and Natural mode, Korean<->English (Qwen3-8B, 4-bit, through a local Ollama) | Apache-2.0 | yes | 500a1f067a |
| facebook/nllb-200-3.3B | Translation baseline in scripts/mt_bench.py only (tokenizer); replaced by MADLAD in C75 | CC BY-NC 4.0 | no — being replaced | 1a07f7d195 |
| KomorebiAI/nllb-200-3.3B-int8-ct2 | Translation baseline in scripts/mt_bench.py only (int8 weights) | CC BY-NC 4.0 (conversion of facebook/nllb-200-3.3B) | no — being replaced | 85ad380f5e |
| inclusionAI/Ming-3.0-tts-0.5B | Voices (English, Korean, acted emotions) | MIT | yes | 7247f4c94a |
| FunAudioLLM/Fun-CosyVoice3-0.5B-2512 | Voices (English, Korean; retry voice) | Apache-2.0 | yes | 29e01c4e8d |
| FunAudioLLM/CosyVoice2-0.5B | Voices (the base Navoiy builds on) | Apache-2.0 | yes | eec1ae6c79 |
| aisha-org/navoiy-tts | Voices (Uzbek) | Apache-2.0 | yes | 9445f41ee5 |
| hexgrad/Kokoro-82M | Preset voices; live English voice | Apache-2.0 | yes | f3ff357179 |
| ResembleAI/chatterbox | Voices (21 other languages, Chatterbox Multilingual) | MIT | yes | 5bb1f6ee58 |
| ResembleAI/chatterbox-turbo | Voices (Chatterbox Turbo) | MIT | yes | 749d1c1a46 |
| Supertone/supertonic-3 | Live Korean voice; Korean fallback voice (ONNX on the CPU) | OpenRAIL-M (commercial use allowed with use-based restrictions, Attachment A; sample code MIT) | yes | 3cadd1ee63 |
| facebook/mms-tts-kor | Not used since C78 (Korean is Supertonic-3); kept for comparisons | CC BY-NC 4.0 | no — being replaced | 1b6491366d |
| facebook/mms-tts-uzb-script_cyrillic | Live Uzbek voice; fallback voice | CC BY-NC 4.0 | no — being replaced | a78336f13a |
| rhasspy/piper-voices | Voices (Kazakh, kk_KZ-issai-high) | MIT repository; ISSAI voice CC BY 4.0 (piper-tts package GPL-3.0, see licensing.md) Kazakh voice by ISSAI (Institute of Smart Systems and AI, Nazarbayev University), CC BY 4.0 |
yes | e7229f7d33 |
| emotion2vec/emotion2vec_plus_large | Emotion tags | FunASR Model Open Source License 1.1 (use, copy, modify and share with attribution) emotion2vec+ large, FunASR (Alibaba Group) |
to confirm | 6c303ba987 |
| facebook/audioseal | Watermark on the dubbed voice (AI-speech mark) | MIT (code and weights; generator_base.pth, detector_base.pth) | yes | 3c19eba533 |
| MIT/ast-finetuned-audioset-10-10-0.4593 | Sounds heard (laughs, sighs) | BSD-3-Clause | yes | f826b80d28 |
AI tools (Claude) were used to write code and documentation; the team reviewed and tested it.