Skip to content

feat(translation): stream target-language translation during chat (#829)

Jérôme Revillard requested to merge feat/stream-translation into main

What & why

Fixes #829 (closed). When the UI language is non-English, the chat answer streamed in English and only flipped to the target language after streaming completed. Root cause: the backend translated the FULL EN answer after the stream (query-routes.js:handleStreamDone).

This MR streams the target-language translation during generation — the user sees the requested language throughout, no flip.

Approach (respects the dedicated translation LLM)

The translation LLM (gemma-3-4b-it / translategemma on vLLM) cannot reliably translate a partial prefix (research in #829 (closed): translation of a prefix ≠ prefix of the translation; simultaneous MT needs a wait-k policy + training). So the backend buffers incoming EN and commits a complete unit (sentence / paragraph boundary) to the translator, then streams its output:

  • extractCommittableUnit() — pure, tested helper (commits at \n\n or . ! ? + close + whitespace; maxBuffer safety fallback).
  • callVllmStream() / translateStream() — vLLM stream:true output parsing + a rolling context window (prior units EN+target) for terminology/pronoun consistency (prompt-based models).
  • query-routes /queries/stream — when STREAMING_TRANSLATION_ENABLED is set and context.language ≠ EN: buffer EN → commit boundary → translateStream → forward translated tokens as chunk events. Flag off / EN → unchanged behavior.

Env-gated (backward-compatible, ansible-managed)

STREAMING_TRANSLATION_ENABLED (default 0): docker-compose backend env + env.j2 (streaming_translation_enabled group_var) + env template. Rollback = unset the flag.

Frontend

No change needed — onChunk appends both EN (flag off) and translated (flag on) chunks; onTranslation retained for the flag-off fallback path.

Testing

  • Boundary helper: 13 unit tests (paragraph/sentence boundaries, maxBuffer fallback).
  • Route test: flag on → translated chunks streamed, no post-stream translation event.
  • 1560 backend tests pass, ESLint + Prettier clean.
  • E2E (real vLLM): to be validated on the El Salvador deployment (set streaming_translation_enabled: "1" in group_vars, redeploy) — that's the only env with the translation LLM running.

Commits

  1. boundary helper + tests (foundation)
  2. streaming-translation core (callVllmStream, translateStream, query-routes rewire)
  3. flag wiring (docker-compose/env.j2/env) + route test + docs

Merge request reports