Skip to content

Chat: answer streams in English; target-language translation only applied after stream completes

Problem

When the UI language is set to a non-English language (e.g. Spanish), the assistant's chat answer streams to the client in English, and only switches to the target language AFTER streaming completes. The user expects the streamed text to be in the requested language throughout (no "English then flip").

Root cause (verified)

components/gov-chat-backend/routes/query-routes.js:

  • L222 — during streaming, EN chunks from chatqna are forwarded to the client verbatim (type: 'chunk').
  • L221 — EN text accumulated in fullResponseText.
  • handleStreamDone (L297-334) — AFTER the stream ends: if context.language ≠ EN, translates the full accumulated EN text via the dedicated translation LLM and emits a single translation event (L316), then done.

Frontend components/gov-chat-frontend/src/components/ChatBotComponent.vue:

  • onChunk (L903) — appends EN while streaming.
  • onTranslation (L940-941) — replaces the whole message with the translation → the "flip" at the end.

So translation is deferred to post-stream. chatqna streams raw EN (genie-ai-overlay/chatqna/genieai_chatqna.py:1075-1109); the dedicated translation LLM is invoked only once, on the complete text.

Architecture constraint

There is a dedicated translation LLM (vLLM vllm-translation-guardrail:9031, VLLM_TRANSLATION_MODEL_ID = gemma-3-4b-it / translategemma-4b-it) that must remain the translation mechanism (English is the RAG source of truth; answers are generated in EN then translated). This issue is about making that translation stream, not replacing the translator.

Research findings

  • The translation models are decoder-only causal → vLLM supports stream: true; the partial/streamed output is coherent (each token is final when emitted). Source: vLLM — Streaming Requests & Realtime API.
  • A translation model cannot reliably translate from partial/incremental input (a growing prefix): the translation of a prefix ≠ the prefix of the translation (cross-lingual word-order), so feeding tokens incrementally produces unstable/rewriting output. True incremental ("simultaneous") translation requires a wait-k read/write policy + a model trained for it (SiMT research field). The generic translator does not self-buffer.
  • Conclusion: the model needs a complete unit (sentence/paragraph) to translate well, but can stream its output once given one.

Proposed solution — streaming translation, buffer-by-paragraph + context window

Instead of forwarding EN chunks and translating the whole thing at the end:

  1. Backend buffers incoming EN chunks until a commit boundary (markdown paragraph \n\n, or sentence . ! ? if the paragraph is long).
  2. On each boundary, call the translation LLM with stream: true on the complete unit, with a rolling context window (a few prior sentences EN+ES) so terminology/pronoun consistency is preserved. Commit/stream only the new unit's translation.
  3. Forward the streamed target-language tokens to the client as chunk events. Remove the post-stream translation event.

Result: the user sees the target language stream paragraph-by-paragraph from the start.

Changes

  • services/translation/gpu-translate-backend.js — add callVllmStream() (POST /v1/chat/completions with stream:true, yield output tokens).
  • services/translation-service.js — add translateStream(unit, sourceLang, targetLang, context).
  • routes/query-routes.js — rewire the stream handler to buffer EN → boundary → translateStream → forward translated chunks; remove the deferred translation in handleStreamDone (L307-323).
  • ChatBotComponent.vue — onChunk already appends (now appends target-language text); onTranslation becomes unused (keep as no-op or remove).
  • Mobile — shares the same backend /queries/stream endpoint → benefits automatically.

Trade-offs / notes

  • Context-window latency: each call carries a small context (a few sentences). vLLM prefix caching caches the repeated context prefix → marginal overhead; the dominant cost is translating the new unit (same as today).
  • Granularity: paragraph (more context, fewer calls) vs sentence (more reactive). Recommend paragraph with sentence fallback for long paragraphs.
  • Boundary detection in markdown: handle lists, code blocks, **bold** (regex on \n\n / . ! ? + a size/timeout fallback so long sentences aren't held too long).
  • Quality vs current: the current translateMarkdown ALSO translates per text-node (segment), so per-unit streaming is not a context regression — the context window provides ≥ current context.

Acceptance criteria

  • With UI language = es, the streamed answer is in Spanish throughout (no English-then-flip).
  • English UI unaffected (no translation path).
  • Translation quality ≥ current (context window preserved).
  • Mobile client benefits too (shared backend).
  • Backend tests for the streaming-translation path.

Affects

Web (gov-chat-frontend) + Mobile (genie_ai_mobile) — both consume /queries/stream.