You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Wikifeeds: Difference between revisions
imported>Quiddity (rm former staff :() |
imported>Alex Paskulin m (→See also: Fix link) |
||
| (3 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
[[File:Wikifeeds simplified example request flow.png|alt=Wikifeeds simplified request flow|thumb|Wikifeeds simplified request flow]] | |||
'''Wikifeeds''' is an API service served via [[RESTBase]]. The functionality provided by this was previously part of [[Mobileapps (service)|Mobile content services (MCS)]]. | '''Wikifeeds''' is an API service served via [[RESTBase]]. The functionality provided by this was previously part of [[Mobileapps (service)|Mobile content services (MCS)]]. | ||
| Line 24: | Line 25: | ||
See docs on the English Wikipedia: https://en.wikipedia.org/api/rest_v1/#/Feed and https://en.wikipedia.org/api/rest_v1/#/Page%20content/get_page_random__format_. | See docs on the English Wikipedia: https://en.wikipedia.org/api/rest_v1/#/Feed and https://en.wikipedia.org/api/rest_v1/#/Page%20content/get_page_random__format_. | ||
Some of the responses for these endpoints are consolidated in RESTBase from a larger number of more narrowly focused responses provided by the wikifeeds service. | {{note | Some of the responses for these endpoints are consolidated in RESTBase from a larger number of more narrowly focused responses provided by the wikifeeds service. This means that a single RESTBase request might end up creating multiple wikifeeds requests that will in turn generate more RESTBase requests}} | ||
== Technology == | == Technology == | ||
| Line 33: | Line 34: | ||
=== Deployment === | === Deployment === | ||
The images that are used in production can be found on the | The images that are used in production can be found on the [https://docker-registry.wikimedia.org/wikimedia/mediawiki-services-wikifeeds/tags/ WMF docker registry]. New images are built, after code is merged to the master branch, automatically by the deployment pipeline. | ||
The production clusters are managed using kubernetes and helm. These are also used for a staging instance as well. The configuration for these can be found in the [[gerrit:plugins/gitiles/operations/deployment-charts/|operations/deployment-charts repo]]. Details for applying those adjustments to the production clusters can be found [[Migrating from scap-helm|here]]. | The production clusters are managed using kubernetes and helm. These are also used for a staging instance as well. The configuration for these can be found in the [[gerrit:plugins/gitiles/operations/deployment-charts/|operations/deployment-charts repo]]. Details for applying those adjustments to the production clusters can be found [[Migrating from scap-helm|here]]. | ||
| Line 65: | Line 66: | ||
The service reports response times via the metrics API provided by [https://github.com/wikimedia/service-runner/blob/master/README.md#metric-reporting service-runner] (as is the standard behavior of services based on the Node.js service template). | The service reports response times via the metrics API provided by [https://github.com/wikimedia/service-runner/blob/master/README.md#metric-reporting service-runner] (as is the standard behavior of services based on the Node.js service template). | ||
== Service level indicators/objectives (SLIs/SLOs) | == Service level indicators/objectives (SLIs/SLOs) == | ||
{| class="wikitable" | {| class="wikitable" | ||
!colspan="2"|SLI: Traffic | !colspan="2"|SLI: Traffic | ||
| Line 97: | Line 98: | ||
| All endpoints || 1/1000 | | All endpoints || 1/1000 | ||
|} | |} | ||
Table patterns from <ref>https://landing.google.com/sre/sre-book/chapters/service-level-objectives/</ref>. | |||
== See also == | == See also == | ||
* [https://en.wikipedia.org/api/rest_v1/#/Feed RESTBase API Docs for "Feeds"] | * [https://en.wikipedia.org/api/rest_v1/#/Feed RESTBase API Docs for "Feeds"] | ||
* [https://grafana.wikimedia.org/d/ | * [https://grafana.wikimedia.org/d/lxZAdAdMk/wikifeeds Grafana dashboard] | ||
* [[mw:Wikifeeds]] | * [[mw:Wikifeeds]] | ||
* [[gerrit:plugins/gitiles/mediawiki/services/wikifeeds|Source repository]] | * [[gerrit:plugins/gitiles/mediawiki/services/wikifeeds|Source repository]] | ||
Latest revision as of 19:29, 13 February 2023
Wikifeeds is an API service served via RESTBase. The functionality provided by this was previously part of Mobile content services (MCS).
Contacts
The service is maintained by the Product Infrastructure team, who can be reached on IRC in #wikimedia-infrastructure connect.
Primary contacts:
- Yiannis Giannelos (
nemo-yiannis) - Mateus Santos (
mateusbs17)
Petr Pchelko (Pchelolo) can also help in the case of operational emergencies related to the RESTBase API gateway.
Overview
Wikifeeds is a node.js webservice supporting the Explore feeds in the official Wikipedia Android and iOS apps.
It creates content that is served via the public REST API feed content endpoints (served by RESTBase):
- /api/rest_v1/feed/featured/{yyyy}/{mm}/{dd}
- /api/rest_v1/feed/announcements
- /api/rest_v1/feed/onthisday/{type}/{mm}/{dd}
- /api/rest_v1/page/random/{format}
See docs on the English Wikipedia: https://en.wikipedia.org/api/rest_v1/#/Feed and https://en.wikipedia.org/api/rest_v1/#/Page%20content/get_page_random__format_.
 | Some of the responses for these endpoints are consolidated in RESTBase from a larger number of more narrowly focused responses provided by the wikifeeds service. This means that a single RESTBase request might end up creating multiple wikifeeds requests that will in turn generate more RESTBase requests |
Technology
Wikifeeds is split off from the main mobileapps/mobile content service. As such it is a nodejs service. It's a service-runner compatible service based on the Node.js service template.
The service is deployed on the WMF services kubernetes cluster using helm. This means that the service is packaged as a docker image. The docker image is built by the Deployment pipeline.
Deployment
The images that are used in production can be found on the WMF docker registry. New images are built, after code is merged to the master branch, automatically by the deployment pipeline.
The production clusters are managed using kubernetes and helm. These are also used for a staging instance as well. The configuration for these can be found in the operations/deployment-charts repo. Details for applying those adjustments to the production clusters can be found here.
Architecture
In a nutshell, the service gathers information about featured content from the templates used to display it on various Wikipedias, and provides it in a standard, structured format.
It provides several endpoints which are in turn consumed by RESTBase to compose larger public endpoint responses:
- For /feed/featured/{yyyy}/{mm}/{dd}:
- /page/featured/{yyyy}/{mm}/{dd}
- /media/image/featured/{yyyy}/{mm}/{dd}
- /page/most-read/{yyyy}/{mm}/{dd}
- /page/news
- /feed/onthisday/selected/{mm}/{dd}
- For /feed/onthisday/all/{mm}{dd}:
- /feed/onthisday/births/{mm}/{dd}
- /feed/onthisday/deaths/{mm}/{dd}
- /feed/onthisday/events/{mm}/{dd}
- /feed/onthisday/holidays/{mm}/{dd}
- /feed/onthisday/selected/{mm}/{dd}
These consolidated responses are composed in RESTBase rather than created directly in the service in order to distribute the load involved in creating them across several workers.
The service also provides a /feed/announcements endpoint that is used to provide content of special announcements for things like campaigns and fundraising. The content of these announcements is currently defined entirely within the service code.
The service also provides a /page/random/title endpoint. RESTBase exposes this as /page/random/{format}. {format} can be something like title, summary, html. So a client can get either just the title or the associated page content.
Metrics
The service reports response times via the metrics API provided by service-runner (as is the standard behavior of services based on the Node.js service template).
Service level indicators/objectives (SLIs/SLOs)
| SLI: Traffic | |
|---|---|
| Endpoint | SLO: Requests per second |
| /page/random | 25 ≤ RPS ≤ 100 |
| All others | 1 ≤ RPS ≤ 10 |
| SLI: Latency | |
|---|---|
| Endpoint | SLO: Max latency (p50) |
| /feed/onthisday/events | 500ms |
| /media/image/featured | 250ms |
| /page/most-read | 200ms |
| All others | 100ms |
| SLI: Server errors (5xx) | |
|---|---|
| Endpoint | SLO: Max 5xx error rate |
| All endpoints | 1/1000 |
Table patterns from [1].