You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

ORES/WMF details: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Awight
(add category)
 
imported>Cwhite
m (T211982)
 
Line 6: Line 6:
We've enabled Redis's builtin [[w:Least Recently Used|LRU]] cache expiration to automatically manage memory.
We've enabled Redis's builtin [[w:Least Recently Used|LRU]] cache expiration to automatically manage memory.


Note that our score caching is not critical to API capacity, it's just a courtesy to users with repetitive workloads.  For example, patrollers using a tool like Huggle will often load the most recent 100 changes, which is a near-instantaneous operation thanks to caching, rather than having to score 100 revisions before responding.  The contribution of our score cache to reducing overall server load is only about 2%, see these graphs: https://grafana.wikimedia.org/dashboard/db/ores?orgId=1&from=1496275200000&to=1512086400000
Note that our score caching is not critical to API capacity, it's just a courtesy to users with repetitive workloads.  For example, patrollers using a tool like Huggle will often load the most recent 100 changes, which is a near-instantaneous operation thanks to caching, rather than having to score 100 revisions before responding.  The contribution of our score cache to reducing overall server load is only about 2%, see these graphs: https://grafana.wikimedia.org/d/HIRrxQ6mk/ores?from=1496275200000&to=1512086400000


== Dictionaries ==
== Dictionaries ==
Some feature extraction modules require aspell or compatible dictionary packages for each language.  We maintain a port of the Indonesian dictionary, here: <nowiki>https://apt.wikimedia.org/wikimedia/pool/main/a/aspell-id/</nowiki>
Some feature extraction modules require aspell or compatible dictionary packages for each language.  We maintain a port of the Indonesian dictionary, here: <nowiki>https://apt.wikimedia.org/wikimedia/pool/main/a/aspell-id/</nowiki>
[[Category:ORES]]
[[Category:ORES]]

Latest revision as of 18:12, 18 February 2022

These are notes specific to the WMF deployment of ORES.

Redis

We're using two redundant Redis nodes in each data center. (Master-replica, or clustered?) Data centers don't exchange any Redis data or scores.

We've enabled Redis's builtin LRU cache expiration to automatically manage memory.

Note that our score caching is not critical to API capacity, it's just a courtesy to users with repetitive workloads. For example, patrollers using a tool like Huggle will often load the most recent 100 changes, which is a near-instantaneous operation thanks to caching, rather than having to score 100 revisions before responding. The contribution of our score cache to reducing overall server load is only about 2%, see these graphs: https://grafana.wikimedia.org/d/HIRrxQ6mk/ores?from=1496275200000&to=1512086400000

Dictionaries

Some feature extraction modules require aspell or compatible dictionary packages for each language. We maintain a port of the Indonesian dictionary, here: https://apt.wikimedia.org/wikimedia/pool/main/a/aspell-id/