Multilingual Language Detection
At South Telecom, I built an XLM-R classifier for language identification. I prepared Vietnamese and mixed‑language samples, then benchmarked accuracy and inference speed across datasets.
Language detection pipeline
The pipeline takes a text sample. It can be a full sentence, a short message, or a mix of Vietnamese and another language.
Normalize equivalent Unicode forms and clean spacing while preserving accents and words that can help identify the language.
Tokenize the prepared text for the XLM-R classifier, which predicts a language label for the sample.
- Role
- AI Engineer · South Telecom
- Period
- Sep – Nov 2025
- Stack
- Python · XLM-R · SentencePiece
Preserving language signals
I normalized Unicode and cleaned text before tokenization. Vietnamese diacritics and words from other languages stay intact. The classifier uses those signals to predict a language label for the whole sample.
Evaluating the classifier
I wrote scripts to measure accuracy against reference labels and compare inference speed across datasets. Short and mixed-language messages are the hardest cases. A mixed sample needs an explicit labeling rule, such as choosing a dominant language or allowing multiple labels, before its accuracy means anything.
Inspect text preprocessing
Choose a sample to see what changes before the text reaches the classifier.
Raw text
Cà phê sáng nay
Prepared text
Cà phê sáng nay
U+0061 + U+0300U+00E0These lines look alike. NFC composes equivalent Unicode sequences while preserving the Vietnamese accents.
Raw text
··Tôi·cần···cập·nhật·ứng·dụng.··
Each dot marks a space.Prepared text
Tôi cần cập nhật ứng dụng.
Text cleaning removes extra spaces at the beginning and end and between words.
Prepared text
Mình sẽ check lại build này.
This input is already normalized. Words such as “check” and “build” stay in the text so the classifier can consider both language signals.