Dubbing is not switched on yet. You can look around and create your account now.

About

A dub that tells you what to check

Every step of automatic dubbing is a model, and errors compound: a misheard word becomes a confident wrong translation, then fluent synthetic speech that says something wrong. DUBBE's answer is a second output next to the dub: a short list of sentences a person should check, shown on the video's timeline.

A capstone project

Applied AI & Business Capstone Design, Inha University × HumbleBeeAI, Fall 2026: Team 1, project C2b, Video Translation and Dubbing.

Code on GitHub

Team

KU
Kalandarov UlugbekIBTTeam Representative / PM
IA
Isokulov AzamatIBTProblem & Business
SM
Samandarova MadinaIBTData & Research, User & Requirements
MJ
Musurmonov JakhongirIBTDemo & Presentation
TR
Tagaev RashidISEAI / Technical Lead
KT
Kholmirzaev TemurjonISESoftware / IT Engineer

Models and licences

Every step is an open model that we run ourselves, on rented GPUs. The live Korean and Uzbek voices (MMS) are the part that is still non-commercial.

Speech recognition

Whisper large-v3 / large-v3-turbo (faster-whisper); Uzbek: rubaistt v2

MIT · Apache 2.0

Voice and background

Demucs (htdemucs)

MIT

Who speaks when

NVIDIA Nemotron 3 Diarization, with TitaNet-large voice fingerprints

OpenMDW 1.1 · CC BY 4.0

Translation

MADLAD-400 3B (int8, CTranslate2), fine-tuned by the team for Uzbek; Korean ↔ English second opinion: Qwen3-8B

Apache-2.0 · data ODC-BY

English and Korean voices

Ming-3.0-tts 0.5B and CosyVoice 3, from Kokoro-82M preset recordings

MIT · Apache 2.0

Uzbek and Kazakh voices

Navoiy TTS (CosyVoice 2); Piper with the ISSAI Kazakh voice

Apache 2.0 · MIT, voice CC BY 4.0

The other languages

Chatterbox Multilingual; MMS as a fallback

MIT · MMS CC BY-NC 4.0

Emotion and sounds heard

emotion2vec+ large (FunASR); AST on AudioSet

FunASR model licence · BSD 3-Clause
Every model, its licence and the exact version we run (29)
ModelStepLicenceCommercial useVersion
Systran/faster-whisper-large-v3Speech recognitionMIT (conversion of openai/whisper-large-v3, Apache-2.0) yes edaa852ec7
mobiuslabsgmbh/faster-whisper-large-v3-turboSpeech recognition, live dubbing, heard-back checkMIT (conversion of openai/whisper-large-v3-turbo, MIT) yes 0a363e9161
islomov/rubaistt_v2_mediumSpeech recognition (Uzbek)Apache-2.0 yes af46011761
htdemucsVoice and background separationMIT (Demucs v4 package and weights, dl.fbaipublicfiles.com) yes 955717e8-8
nvidia/Nemotron-3-DiarizationWho speaks whenOpenMDW-1.1 yes f667ed73ae
nvidia/speakerverification_en_titanet_largeVoice fingerprints (speakers)CC BY 4.0
TitaNet-large by NVIDIA, CC BY 4.0
yes 0dc382f401
microsoft/wavlm-base-plus-svVoice fingerprints (fallback without the NeMo environment)none on the card (upstream microsoft/unilm repository is MIT); team rule 2026-11-11, no licence = MIT yes feb593a6c2
Nextcloud-AI/madlad400-3b-mt-ct2-int8Translation (MADLAD-400 3B, int8 CTranslate2 conversion of google/madlad400-3b-mt)Apache-2.0 yes aa32bbdeba
dubbe-ai/madlad400-3b-uz-ct2-int8Translation, MADLAD-400 3B fine-tuned by the team for Uzbek (C81; LoRA merged into google/madlad400-3b-mt, int8 CTranslate2). Local folder, loaded when DUBBE_MADLAD points at it; not uploadedApache-2.0 (derivative of google/madlad400-3b-mt); training data OPUS NLLB v1, ODC-BY 1.0
Fine-tuned on OPUS NLLB v1 (data mined by Meta AI, released by AllenAI; ODC-BY 1.0; opus.nlpl.eu)
yes sha256-427
Qwen/Qwen3-ASR-1.7B-hfSecond listen (a second recogniser next to Whisper; flags sentences the two hear differently)Apache-2.0 yes bcd2b5b7f3
google/metricx-24-hybrid-large-v2p6-bfloat16Translation quality without a reference (MetricX-24, QE mode)Apache-2.0 yes febb720e29
google/mt5-xlTokenizer of MetricX-24 (only spiece.model and its configs are downloaded)Apache-2.0 yes 63fc6450d8
qwen3:8bSecond opinion and Natural mode, Korean<->English (Qwen3-8B, 4-bit, through a local Ollama)Apache-2.0 yes 500a1f067a
facebook/nllb-200-3.3BTranslation baseline in scripts/mt_bench.py only (tokenizer); replaced by MADLAD in C75CC BY-NC 4.0 no — being replaced 1a07f7d195
KomorebiAI/nllb-200-3.3B-int8-ct2Translation baseline in scripts/mt_bench.py only (int8 weights)CC BY-NC 4.0 (conversion of facebook/nllb-200-3.3B) no — being replaced 85ad380f5e
inclusionAI/Ming-3.0-tts-0.5BVoices (English, Korean, acted emotions)MIT yes 7247f4c94a
FunAudioLLM/Fun-CosyVoice3-0.5B-2512Voices (English, Korean; retry voice)Apache-2.0 yes 29e01c4e8d
FunAudioLLM/CosyVoice2-0.5BVoices (the base Navoiy builds on)Apache-2.0 yes eec1ae6c79
aisha-org/navoiy-ttsVoices (Uzbek)Apache-2.0 yes 9445f41ee5
hexgrad/Kokoro-82MPreset voices; live English voiceApache-2.0 yes f3ff357179
ResembleAI/chatterboxVoices (21 other languages, Chatterbox Multilingual)MIT yes 5bb1f6ee58
ResembleAI/chatterbox-turboVoices (Chatterbox Turbo)MIT yes 749d1c1a46
Supertone/supertonic-3Live Korean voice; Korean fallback voice (ONNX on the CPU)OpenRAIL-M (commercial use allowed with use-based restrictions, Attachment A; sample code MIT) yes 3cadd1ee63
facebook/mms-tts-korNot used since C78 (Korean is Supertonic-3); kept for comparisonsCC BY-NC 4.0 no — being replaced 1b6491366d
facebook/mms-tts-uzb-script_cyrillicLive Uzbek voice; fallback voiceCC BY-NC 4.0 no — being replaced a78336f13a
rhasspy/piper-voicesVoices (Kazakh, kk_KZ-issai-high)MIT repository; ISSAI voice CC BY 4.0 (piper-tts package GPL-3.0, see licensing.md)
Kazakh voice by ISSAI (Institute of Smart Systems and AI, Nazarbayev University), CC BY 4.0
yes e7229f7d33
emotion2vec/emotion2vec_plus_largeEmotion tagsFunASR Model Open Source License 1.1 (use, copy, modify and share with attribution)
emotion2vec+ large, FunASR (Alibaba Group)
to confirm 6c303ba987
facebook/audiosealWatermark on the dubbed voice (AI-speech mark)MIT (code and weights; generator_base.pth, detector_base.pth) yes 3c19eba533
MIT/ast-finetuned-audioset-10-10-0.4593Sounds heard (laughs, sighs)BSD-3-Clause yes f826b80d28

AI tools (Claude) were used to write code and documentation; the team reviewed and tested it.