Skip to content

This commit fixes a critical bug in the `2-update_embeddings.py` scri…

Edouard Legoupil requested to merge fix-embedding-script-self-contained into main_dev

Created by: Edouard-Legoupil

…pt where database transactions were failing silently, resulting in no embeddings being saved. The fix is self-contained within the script to avoid any side effects on shared utility functions used by the API.

The root cause was improper SQLAlchemy session handling in a concurrent environment. The script was sharing a single session across multiple threads, and the asyncio.run() call within a synchronous loop was interfering with the transaction management.

The solution is as follows:

  1. Self-Contained Worker Function: A new function, process_reference_safely, has been created inside the script. This function encapsulates all the logic for processing a single reference.
  2. Isolated Session Management: Crucially, this worker function is responsible for creating and managing its own database session and transaction. It is passed the SessionLocal factory and creates a new session within a with block, ensuring that each thread of execution has its own isolated transaction.
  3. Thread-Safe Concurrency: The main body of the script now uses a concurrent.futures.ThreadPoolExecutor to run the process_reference_safely worker for each reference in parallel.
  4. No Shared Code Modification: The shared backend/utils/embedding_utils.py file has been restored to its original state and is untouched by this change, completely isolating the fix and guaranteeing no regressions in the API.

This approach is simpler, safer, and correctly handles database transactions in a multi-threaded context, ensuring that the embedding data is now reliably and correctly saved.

  • Does my code meet the quality standards for releasing packages?
  • Does the reviewer have all the information to validate the features/issues without too much research?
  • Does the customer who will validate the associated tickets have the information to do so without wasting time?

Issues to validate to close :

  • issue #

Processed issues to keep open or in progress:

  • issue #

Checklist:

  • Does the package check go local?
  • Does the CI pass?
  • Are the added / fixed features documented, tested?
  • Are the added features / solved problems briefly presented in the PR message?
  • Are the changes related to tickets / issues that I have listed in the commits and in the PR itself?
  • Are the tickets in "review" mode in the Project Tracking Board?
  • Does each ticket, if it is to be closed after acceptance of the PR, contain a comment that tells how to validate it?

Merge request reports