Skip to content

fix(chatqna): strip leaked |<-MSG->| markers from streaming SSE

Jérôme Revillard requested to merge fix/streaming-msg-separator-leak into main

Problem

The streaming path forwarded LLM token chunks verbatim, so the internal |<-MSG->| history delimiters and USER:/ASSISTANT: role markers the LLM sometimes echoes back leaked through the SSE stream to the frontend.

The non-streaming path already strips these from the assembled llm_response, but the streaming branch (_stream_with_metadata) returns early and bypassed that stripping. El Salvador (streaming + streaming-translation) was affected — markers reached the backend/frontend unprocessed.

Streaming:   LLM chunks → SSE → frontend (RAW, not stripped) → user sees |<-MSG->|
Non-stream:  LLM response → stripping (line ~1918) → frontend → clean

Fix

Apply the same stripping inside _stream_with_metadata (the single choke point every streamed token passes through). Because a marker can be split across token chunks (e.g. |<-M + SG->|), content is buffered:

  • only the settled head is re-emitted
  • a tail that could be the start of a marker is held back until the marker completes (and is stripped) or enough non-marker text arrives
  • the role-marker regex tolerates leading whitespace so a marker split from its preceding separator across chunks (separator → \\n in one chunk, USER: arriving next) is still stripped
  • unparseable chunks are forwarded verbatim so an unexpected orchestrator format never breaks the stream

Verification

  • 12 new TestStreamWithMetadata cases: separator/role in single chunk + across chunks, role split from separator, normal-content preservation, lone-pipe preservation, partial-marker flush, verbatim passthrough, unicode, realistic full-history echo
  • Adversarial sim (34 pseudo-random 1-5 char chunks through a full history echo): 0 marker leakage, all real content preserved
  • Full OPEA suite: 425 passed
  • Ruff lint + format: clean

Merge request reports