Skip to content

This commit enhances the `2-update_embeddings.py` script by adding a …

Edouard Legoupil requested to merge feature-deduplicate-and-find-references into main_dev

Created by: Edouard-Legoupil

…new --test-scrap command-line option. When this option is used, the script will:

  • Iterate through all the reference URLs in the database.
  • Attempt to scrape each URL to check for accessibility and content.
  • Log any URLs that fail to scrape, along with the corresponding error message, to a timestamped CSV file in the log/ directory.

This provides a convenient way to test the scrapability of all reference URLs without running the full embedding update process, making it easier to identify and debug issues with broken or inaccessible links.

  • Does my code meet the quality standards for releasing packages?
  • Does the reviewer have all the information to validate the features/issues without too much research?
  • Does the customer who will validate the associated tickets have the information to do so without wasting time?

Issues to validate to close :

  • issue #

Processed issues to keep open or in progress:

  • issue #

Checklist:

  • Does the package check go local?
  • Does the CI pass?
  • Are the added / fixed features documented, tested?
  • Are the added features / solved problems briefly presented in the PR message?
  • Are the changes related to tickets / issues that I have listed in the commits and in the PR itself?
  • Are the tickets in "review" mode in the Project Tracking Board?
  • Does each ticket, if it is to be closed after acceptance of the PR, contain a comment that tells how to validate it?

Merge request reports