You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
Category:Data pipelines
Jump to navigation
Jump to search
Documentation of data ingestion and processing pipelines.
- Includes documentation describing how specific datasets are derived or computed, for example: MediaWiki history computation (ingestion from DB, history rebuilding, computation of metrics, extraction onto other systems, ad-hoc querying).
- Does not include documentation for the data platform infrastructure or system components that implement a given data pipeline, for example: Airflow, Gobblin.
Pages in category "Data pipelines"
The following 13 pages are in this category, out of 13 total.
D
- Data Platform/Systems/Edit data loading
- Data Platform/Systems/Edit history administration
- Data Platform/Systems/Edit serving layer
- Data Platform/Systems/Hadoop Event Ingestion Lifecycle
- Data Platform/Systems/Mediawiki history reduced algorithm
- Data Platform/Systems/Mediawiki History Snapshot Check
- Data Platform/Systems/Page and user history reconstruction
- Data Platform/Systems/Page and user history reconstruction algorithm
- Data Platform/Systems/Revision augmentation and denormalization
- Data Platform/Systems/Wikistats/Traffic