AI work

Multilingual Language Detection

At South Telecom, I built an XLM-R classifier for language identification. I prepared Vietnamese and mixed‑language samples, then benchmarked accuracy and inference speed across datasets.

Language detection pipeline

The pipeline takes a text sample. It can be a full sentence, a short message, or a mix of Vietnamese and another language.

Normalize equivalent Unicode forms and clean spacing while preserving accents and words that can help identify the language.

Tokenize the prepared text for the XLM-R classifier, which predicts a language label for the sample.

Role
AI Engineer · South Telecom
Period
Sep – Nov 2025
Stack
Python · XLM-R · SentencePiece

Preserving language signals

I normalized Unicode and cleaned text before tokenization. Vietnamese diacritics and words from other languages stay intact. The classifier uses those signals to predict a language label for the whole sample.

Evaluating the classifier

I wrote scripts to measure accuracy against reference labels and compare inference speed across datasets. Short and mixed-language messages are the hardest cases. A mixed sample needs an explicit labeling rule, such as choosing a dominant language or allowing multiple labels, before its accuracy means anything.

Inspect text preprocessing

Choose a sample to see what changes before the text reaches the classifier.

Raw text

Cà phê sáng nay

Prepared text

Cà phê sáng nay

NFC compositionU+0061 + U+0300U+00E0

These lines look alike. NFC composes equivalent Unicode sequences while preserving the Vietnamese accents.

Raw text

··Tôi·cần···cập·nhật·ứng·dụng.··

Each dot marks a space.

Prepared text

Tôi cần cập nhật ứng dụng.

Text cleaning removes extra spaces at the beginning and end and between words.

Prepared text

Mình sẽ check lại build này.

This input is already normalized. Words such as “check” and “build” stay in the text so the classifier can consider both language signals.