Skip to content

feat(sst): Server-Side Tools — web search, stream ingestor, admin UI

David FORDEN requested to merge feat/sst into main

Context

The RAG pipeline is a fixed linear chain (query → embed → retrieve → rerank → LLM). Users and administrators need the system to search the live web, ingest streaming content (RSS), and manage these tools with proper access controls — without waiting for the OPEA 1.5 upgrade.

This MR delivers the Server-Side Tools (SST) pillar of the agentic-enablement initiative: the first set of tools that augment the RAG pipeline beyond static document retrieval.

Related to #844 (closed)

Changes

1. SearXNG web search integration (Python)

  • genie-ai-overlay/workflows/tools/web_search.py — SearXNG search wrapper
  • genie-ai-overlay/workflows/tools/fusion.py — merges tool outputs into RAG context
  • configs/searxng/settings.yml — self-hosted SearXNG instance config
  • Governance scaffolding: PII redaction, rate limiting, circuit breaker
  • ChatQnA orchestrator wired for tool dispatch (genieai_chatqna.py)

2. Stream ingestor service (Python, new container)

  • genie-ai-overlay/stream_ingestor/ — polling-based RSS/Atom microservice
  • Per-feed exponential backoff circuit breaker
  • Dedup via entry ID + SHA256; automatic retraction of expired chunks
  • Dataprep accepts sourceType/feedId/expiresAt metadata

3. Admin backend (Node.js)

  • tools-service.js / tools-routes.js — feed CRUD + SearXNG test proxy
  • DB migration 005-create-feeds-collection.js
  • Keycloak proxy: assignRealmRole / removeRealmRole for tools-admin, tools-reader
  • Admin routes: POST /users/:userKey/roles, DELETE /users/:userKey/roles/:roleName

4. Admin frontend (Vue 3)

  • AdminToolsView.vue — feed list/add/edit/delete + SearXNG test panel
  • Vuex tools.js module + /admin/tools route
  • Role assignment modal in AdminDashboard user table
  • Citation rendering: [1], [2, 3] → clickable superscript links in chat

5. Infrastructure

  • docker-compose.yaml — stream-ingestor + SearXNG service definitions
  • .gitlab-ci.yml — build/scan/promote jobs (follows ADR 0001 pattern)
  • deploy/ansible/deploy.yml — stream-ingestor deployment task
  • configs/keycloak/genie-realm.yaml — tools-admin, tools-reader roles

6. Documentation

  • ADR 0002 (docs/adr/0002-server-side-tools-architecture.md) — 7 architecture decisions covering tool execution model, SearXNG, stream ingestor, admin governance, role-based access, citation rendering, CI/CD
  • Test scaffolding for governance and SST tools
  • Updated PRD with implementation status

Design decisions (see ADR 0002)

  • Pipeline fusion, not agent loop — tools run deterministically, controlled by admin config, not by the LLM. Forward-compatible with OPEA 1.5 AgentQnA.
  • SearXNG over commercial APIs — sovereign, self-hosted, no API keys or billing.
  • Polling over webhooks — RSS is inherently pull-based; simpler, no public endpoints needed.
  • BFF governance — all tool config flows through the Node.js backend (consistent auth, audit trail).
  • Separate roles — tools-admin/tools-reader instead of overloading the existing admin role.

Validation

  • SearXNG wrapper tested locally against self-hosted instance
  • Stream ingestor polling loop + circuit breaker verified with sample RSS feeds
  • Frontend tools UI renders feed CRUD operations and SearXNG test results
  • Citation regex tested against LLM output patterns ([1], [1, 2], [1, 2, 3])
  • CI jobs follow existing build_template/scan_template/promote_template (ADR 0001)
Edited by Adem Mcharek

Merge request reports