Oʻlchov natijalari
We have no customers yet, so instead of logos: what we measured, on which clips, and when. Including what does not work well. Test setup: one laptop GPU (RTX 4060, 8 GB). Full log in the repository.
Test data
| FLEURS, read sentences in Korean and 20 more languages | public dataset (CC BY 4.0), with reference transcripts and translations |
| KsponSpeech, 150 clips of conversational Korean (14 min) | research dataset with transcripts — local test only |
| 529 Korean–English sentence pairs: TED talks, everyday expressions, FLEURS | human translations (OPUS TED2020, AI Hub, FLEURS) — local test only |
| CREMA-D, 160 acted English sentences | emotion labels, for the emotion model |
| A two-person dialogue, 36 s | made with two synthetic voices — no real people |
| TV clip, Korean, 1:30 · web drama, Korean, 25 min · animated lesson, English, 5:26 | copyrighted — local tests only, never published |
What works
| Tekshirish | Result | Date |
|---|---|---|
| Speech recognition, Korean | 9.1 % character error rate on 22 minutes of read speech (one pass, like a real video); 11.5 % on conversation | 2026-10-16 |
| Translation now: MADLAD-400 3B (allows commercial use), 100 FLEURS sentences per direction | chrF++ 56.2 Korean → English, 57.2 Uzbek → English, 42.0 English → Uzbek; on a news clip 5 tadan 3 meaning traps right (NLLB 0 of 5; Qwen3, the Korean ↔ English second opinion, 5 of 5) | 2026-11-11 |
| Translation before (NLLB-200 3.3B, non-commercial), 529 sentence pairs | chrF++ 52.3 Korean → English and 32.8 English → Korean (49.3 and 29.4 with the smaller model it replaced); fixed "Newt Churchill" → Gingrich, "below sea level" → below freezing | 2026-10-16 |
| Soʻzlovchilar ajratiladi | two synthetic voices, a man and a woman: 13 tadan 13 sentences right; FLEURS readers: the two groups matched the dataset's male / female labels in 15 of 15 sentences | 2026-10-16 |
| A speaker's own voice, cloned from the video with consent | speaker similarity to the person 0.85 (built-in voice: 0.66) | 2026-10-15 |
| Korean voices | 26: the built-in voice and 25 English presets speaking Korean — Whisper understood 20 of them as well as the built-in voice; a two-person Korean dub, one voice per speaker: 0.6 % character errors | 2026-10-16 |
| Voice running on after the sentence | French dub, two sentences ended in made-up words: 11.0 % → 2.1 % character errors after DUBBE cuts at the sentence's last word; the af_alloy voice in Korean 29.3 % → 8.1 %, so it is back in every language | 2026-10-19 |
| Timing drift, three clips | largest delay 0.00 s (the lesson needed the picture slow-down option; 0.37 s without) | 2026-10-06 |
| Long translations (lesson) | strongest voice speed-up 1.30× → 1.18×; sentences that did not fit 16 → 0 | 2026-10-06 |
| Background sound | a phone ringing kept its level (−0.0 dB); 0 words of the original voice left over | 2026-10-06 |
| Voice choice by pitch | matched the speaker's gender for 95 % of Korean and 95 % of English test speakers | 2026-10-06 |
| Speed | 29 s of processing per video minute into English (2:43 clip); 77 s into Korean (5:26 clip) | 2026-10-16 |
The other 20 languages
FLEURS read speech only — no conversation, no native listener yet. Recognition: 20 sentences in one pass, with the Whisper model that made fewer errors in that language. Translation: 100 sentences, chrF++ (higher is better). Voice: an English preset speaking 6 sentences, Whisper listening back.
| Til | Recognition errors | Translation from English | into English | Voice errors |
|---|---|---|---|---|
| Arab | 4.4 % | 53 | 65 | 2.2 % |
| Xitoy | 8.8 % | 23* | 55 | 6.9 % |
| Daniya | 3.2 % | 68 | 71 | 2.6 % |
| Niderland | 3.2 % | 54 | 58 | 0.2 % |
| Fin | 4.5 % | 52 | 60 | 1.4 % |
| Fransuz | 4.1 % | 70 | 68 | 0.7 % |
| Nemis | 1.8 % | 63 | 70 | 2.5 % |
| Yunon | 3.2 % | 52 | 62 | 6.0 % |
| Hind | 13.0 % | 58 | 67 | 6.0 % |
| Italyan | 1.4 % | 59 | 64 | 0.0 % |
| Yapon | 6.8 % | 24* | 56 | 9.6 % |
| Malay | 10.1 % | 66 | 68 | 2.9 % |
| Norveg | 3.6 % | 57 | 65 | 3.1 % |
| Polyak | 2.4 % | 47 | 57 | 5.4 % |
| Portugal | 2.2 % | 69 | 72 | 4.4 % |
| Rus | 3.8 % | 56 | 64 | 1.9 % |
| Ispan | 2.4 % | 55 | 60 | 3.7 % |
| Suaxili | 26.3 % | 60 | 67 | 6.6 % |
| Shved | 2.9 % | 68 | 72 | 0.6 % |
| Turk | 17.2 % | 58 | 65 | 4.8 % |
* chrF++ is not comparable for Japanese and Chinese (no spaces between words). Hebrew was left out: recognition 29 % and voices 38–47 % character errors. French, Hindi and Swahili use only the English presets — their built-in voice failed.
Where errors come from (FLEURS, Korean → English, with the earlier translation model)
We replaced one step at a time with a correct human reference:
- Translation score (chrF): 49.4 with our transcript, 60.1 with a perfect transcript — recognition errors cost about 11 points.
- The translator itself was the larger gap — the reason it was replaced by the larger NLLB model above.
What does not work well yet
- Flags miss confident mistakes. On FLEURS, 3 of 15 sentences were flagged and 2 of them were wrong — but about 8 of 15 were wrong by the author's reading (for example "Ghana" for Canaan). A test with independent raters is planned.
- Short spoken lines are still mistranslated ("That would be great" → 정말 멋져요): sentences are translated one by one, without the conversation around them.
- Similar voices are taken for one speaker — two men in a Korean TV scene became one. Move the sentences in the editor.
- Emotion: the emotion model is right 82 % of the time when it tags (acted English), but finds only a third of the emotional sentences, is not measured on Korean, and tagged 2 of 7 calm sentences of a synthetic male voice [angry]. The [excited] / [angry] effects did not change what an emotion model hears in an English dub (5 of 44 sentences with and without), and Chatterbox's emotion strength setting changed nothing measurable in Korean.
- Recognition is weak for Swahili (26 %), Turkish (17 %) and Hindi (13 %) — fine as target languages, poor as sources.
- The voice choice failed on an agitated multi-speaker TV scene — the editor lets you change it.
- Expressive tags: [whisper], [excited], [sad], [angry] and [question] are made by DUBBE for every voice; for [question] English uses the Chatterbox voice, whose end rose in 11 of 12 test questions. A sound classifier heard the English [cough], [gasp], [laugh] and [sigh] only weakly, and the other languages' voices make none of them.
- No fluent speaker has rated any target language yet; not yet tested on a second computer.