You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Add Link

From Wikitech-static
Revision as of 13:18, 15 October 2020 by imported>Kosta Harlan (→‎12 - 16 October 2020)
Jump to navigation Jump to search

This page contains information about the infrastructure used for the Add a Link structured task project (task T252822)

High-level summary

  1. The Link Recommendation Service is accessible via HTTP (see task T258978), it responds to a POST request containing wikitext of an article (or possibly Parsoid HTML, to be determined) and responds with a structured response of link recommendations for the article. It does not have caching or storage; the client (MediaWiki) is responsible for doing that task T261411.
  2. The search index stores metadata about which articles have link recommendations via a field we set per article
  3. A MySQL table per wiki is used for caching the actual link recommendations (format to be determined task T261411)
  4. A maintenance script regularly generates link recommendations by iterating over each Search/articletopic and calling the Link Recommendation Service
    • the maintenance script caches the results in the MySQL table, then sends an event to Event_Platform/EventGate, where the Search pipeline ensures that the index is updated with the links/nolinks metadata for the article.
    • on page edit and deletion, link recommendations are regenerated via the job queue and the same code and APIs that are utilized in the maintenance script

Open questions

  1. Store the link recommendations in WANObjectCache or in a MySQL table? task T261411(needs SRE/DBA input)
  2. How to get a MySQL database from stat* server to a production MySQL instance (SRE/Analytics)
  3. Do we want to use the job queue to regenerate link recommendations (Growth)

Resolved questions / decisions

  • 15 October: use wikitext for training model, generating dictionary data, and as input to the mwaddlink query service. Will search for phrases in VE's editable content surface rather than attempt to apply offsets from wikitext / parsoid HTML.

Updates

12 - 16 October 2020

  • Growth / Research: Parsoid HTML vs wikitext, repo structure, MySQL vs SQLite, misc other things
  • Growth: Engineers meet to discuss schedule, order of tasks, etc

5 - 9 October 2020

  • Growth / Editing: Exploring ways to bring link recommendation data into VisualEditor
  • Growth / Research: Discussing repository structures in preparation for deployment pipeline setup
  • Growth / SRE / Research: Discussing how to get mwaddlink-query / mwaddlink into production

Teams / Contact

Background reading