A Multilingual RAG Chatbot, and the Cost of Pivoting Through English

What it is

A support chatbot that answers questions from a knowledge base of about four thousand question-and-answer pairs, in whatever language the question was asked in. You type; it finds the handful of entries that are actually relevant; a model writes an answer from those entries rather than from memory; the answer comes back in your language.

The part I cared about was the retrieval. A model asked a support question without context will answer confidently and sometimes wrongly. Giving it the five most relevant entries first changes the job from "recall" to "summarize what is in front of you", which is a job it is far better at.

How it works

One request runs through four services.

Language detection and translation come first: the question is detected and translated into English. The English text is embedded and used to search a FAISS index built over the FAQ pairs, which returns the five nearest entries. Those entries and the question go to Gemini 2.5 Flash, which writes an answer grounded in that context. The answer is translated back into the language of the original question.

Embeddings default to a local sentence-transformer model rather than a paid API, with OpenAI embeddings available behind a flag. That default matters more than it sounds: building and rebuilding an index over four thousand entries while experimenting is free, so I actually experimented.

What works

• Grounding. Answers come from retrieved entries, so the failure mode moves from invented facts to "the right entry was not retrieved" — a failure you can inspect and fix.
• Five entries, not one. Returning the single best match makes the bot brittle when a question spans two topics. Five gives the model enough to synthesize an answer without burying it.
• A local embedding model. No credits, no rate limits, no network round trip in the hot path of building the index.
• Separated services. Translation, vector search and generation are each their own module with their own tests, so a failure is attributable to one of them rather than to "the chatbot".

Where it falls short

Everything pivots through English. This is the design decision the project is built on, and it is the one I would revisit. A question in Urdu is translated to English, matched against English entries, answered in English, and translated back. That is two machine translations per answer, each able to lose the precision that made the question specific — and the retrieval step sees the translation, not the question. An idiom or a product name that translates loosely can miss the right entry entirely. Multilingual embeddings would let the search see the original question, and the knowledge base could be indexed once per language.

Retrieval is fixed at five entries with no threshold. There is no notion of "nothing here is relevant", so a question outside the knowledge base still retrieves five entries and still gets a confident answer written from them. A similarity floor, and an honest "I do not have anything on that", would matter more than any prompt tuning.

No evaluation. There are tests that each service runs, but nothing measures answer quality: no set of questions with expected entries, no retrieval accuracy figure. I know it works because I tried it, which is exactly the standard I criticized in another write-up.

Streamlit is the whole front end. Fine for showing the idea, not a product: no session persistence, no feedback signal, no way for a reader to mark an answer wrong.

What I would do next

Index with a multilingual embedding model and drop the translation round trip for retrieval. Add a similarity threshold and a refusal path. Then build the smallest honest evaluation: fifty real questions, the entry each should retrieve, and a number that has to go up before I change anything else.