Many of the teams we work with write their policies in English and hear questions in other languages. A university’s programme catalogue is in English. The applicant asks about it in Arabic. A school’s handbook is in English, and parents write in French.
Pragence handles this in two separate parts. One part decides which language the reply is written in. The other part makes sure a question in one language can still find a passage written in another. This post explains both, and the Arabic keyword option a deployment can switch on.
Part one: the reply language
The end-user chat API accepts a reply language with each message. Seven are supported: English, Arabic, Urdu, Hindi, Chinese, French and Spanish. The portal sends the language the person picked, and so can your own app through the JavaScript chat client.
When a language is set, Pragence adds an instruction to the assistant: reply in that language unless the person explicitly asks for another one. The same instruction tells it to preserve the meaning of published requirements and policies and not to invent translations of facts. An entry deadline stays the same date. A fee stays the same amount.
The interfaces follow the same seven languages. The portal has a language menu in its header. The website widget’s own labels, such as the button to change language, are translated into each of them, and visitors can switch the widget’s language. In the widget, the language of the reply itself comes from multilingual mode, described below. Arabic and Urdu are written right to left, so the chat window takes its direction from the language, and each message sets its direction from its own text. A reply in Arabic that quotes an English programme name still reads correctly.
Part two: finding English sources with a question in another language
Pragence searches in two ways at once. Vector search compares meanings. Keyword search, using a ranking method called BM25, compares exact words. They behave very differently across languages.
Vector search uses multilingual embeddings, which place text from different languages in one shared space. A question in Arabic lands near the English passage that answers it. No translation is needed.
Keyword search cannot cross languages. An Arabic word never matches an English word. So when a project’s multilingual mode is on, Pragence adds one step at the start of each question. A language model:
- detects the language of the question and returns its language code;
- translates the question into a clean English search query, keeping proper nouns, programme names and acronyms unchanged.
The English query goes to keyword search. The original question goes to vector search. The two result lists are merged as usual.
The step also handles questions typed in Latin letters, such as Arabic written as Arabizi, Hindi as Hinglish or Chinese as pinyin.
When the detected language is not English, the answer is generated in that language, and the citation markers [1], [2] and proper nouns are kept unchanged.
If the detection step fails for any reason, Pragence falls back to the original question. Search is never blocked by a translation error.
Turning it on
Multilingual mode adds one model call per question, so it is a choice. A project turns it on or off with a multilingual setting in its assistant configuration. Projects without that setting follow the deployment-wide ENABLE_MULTILINGUAL value, which is off by default.
This design assumes your sources are mostly in English, because the keyword query is translated into English. Vector search still works for sources in other languages.
The Arabic keyword option
With the standard configuration, keyword search stems English words and leaves Arabic words untouched. Stemming reduces words to a common root so different forms match. Without it, Arabic words only match letter for letter. أحمد does not match احمد, because of the hamza on the first letter. الجامعة (“the university”) does not match جامعة (“university”), because of the attached article.
The database schema includes a second search configuration, pragence_en_ar. It treats English exactly as before. Words in other scripts go through PostgreSQL’s built-in Arabic stemmer, which normalises the hamza and alef forms and strips common prefixes and suffixes. It needs no extra database extension.
This option is off by default. A deployment can switch it on by rebuilding the keyword index with the new configuration and setting the matching value:
uv run python -m scripts.rebuild_bm25_index --text-config pragence_en_ar
BM25_TEXT_CONFIG=pragence_en_ar
The rebuild replaces the index in one transaction, so a failure leaves the old index in place. It blocks writes to the passage table while it runs, so schedule it for a quiet period. The configuration check reports a mismatch if the index and the setting disagree.
It is most useful when you have Arabic source documents, or when people search with Arabic words that appear in your sources, such as names.
What we have checked, and what we have not
In a live test run on 5 October 2026 with synthetic documents, a housing question asked in English, French, Spanish, Arabic, Hindi, Chinese, Yoruba and Hausa returned the same monthly amount from the same source document. The detected language matched each question.
That test checks that the right fact and source come back and that the reply uses the right script. It does not measure how natural the writing is, how well dialects are covered, or whether specialist terms are translated the way your institution would translate them. Before you rely on a language, have a fluent speaker on your team read real answers to real questions.