You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

User:Luke Bowmaker/Data Value Streams: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Luke Bowmaker
 
imported>Luke Bowmaker
Line 1: Line 1:
== Data Value Streams ==
== Data Value Streams ==


* Add values streams: Mission objectives
* TO DO: Populate table
* Add rituals
{| class="wikitable"
|+
|-
!Value Stream!!Objectives!!PM
!EM
!Team
!Board
!Roadmap Link
|-
|'''Event Streams'''||
* Conclude the work from the Event Stream Experiments
* Deliver a consolidated, enriched and ordered stream that is available to the community
* Deliver a way for internal teams to query the current state of MediaWiki with a delay of 3-4 hours, removing the reliance on the monthly dumps
* Deploy Flink to the new DSE k8s cluster as an experimental/development environment
* Deploy Flink to a production multi-dc environment
* Build tooling to support Engineers who want to build event driven services
* Build event driven data integration services that allow teams to be agnostic of the underlying database architecture
* Build a current state store to allow bootstrapping of services and a view of the current state of MediaWiki
|Luke
|Will
|
|[[phab:tag/generated_data_platform/|#generated_data_platform]]
|https://miro.com/app/board/uXjVOtmVf40=/
|-
|'''Data Pipelines & Services'''||
* EMIL - to add infra part...
* Support the Structured Data teams implementation of SDAW grant work: Section Topics Data Pipeline (Q1) and Section Level Image Suggestions (Q2)
|Emil (Infra)/Luke (Services)
|
|
|Need to consolidate or just use DE board
|
|-
|'''RESTBase Deprecation'''||
* Port API Backends used by Visual Editor to CORE
* Migrate AQS 2.0 to k8s production
* Review services to understand required changes
* Investigate and understand access control needs
* Define a RESTBase end of life plan and fully deprecate RESTBase
|Luke
|Atieno
|
|[[phab:tag/api_platform/|#api_platform]]
|https://miro.com/app/board/uXjVOuAghYU=/
|-
|Shared Data Infrastructure|| ||Emil
|
|
|
|
|-
|Metrics Platform|| ||Emil
|
|
|
|
|-
|API Platform|| ||Desiree (Veronica)
|
|
|
|
|-
|Community Datasets (Dumps)|| ||Emil?
|
|
|
|
|-
|'''Platform Engineering Reliability'''||
* Upgrade Thumbor to Py3
* Deploy Thumbor to Commons Beta and test
* Deploy Thumbor to k8s
* Tune Thumbor for prod traffic
* Deprecate legacy Thumbor
|Desiree
|Mat
|
|[[phab:tag/thumbor/|#thumbor]]
|https://miro.com/app/board/uXjVOtKCKBc=/
|}
 
 
==Rituals==
 
*3 week sprints
*Daily stand-ups (except Friday)
*Retro after every sprint

Revision as of 14:49, 6 July 2022

Data Value Streams

  • TO DO: Populate table
Value Stream Objectives PM EM Team Board Roadmap Link
Event Streams
  • Conclude the work from the Event Stream Experiments
  • Deliver a consolidated, enriched and ordered stream that is available to the community
  • Deliver a way for internal teams to query the current state of MediaWiki with a delay of 3-4 hours, removing the reliance on the monthly dumps
  • Deploy Flink to the new DSE k8s cluster as an experimental/development environment
  • Deploy Flink to a production multi-dc environment
  • Build tooling to support Engineers who want to build event driven services
  • Build event driven data integration services that allow teams to be agnostic of the underlying database architecture
  • Build a current state store to allow bootstrapping of services and a view of the current state of MediaWiki
Luke Will #generated_data_platform https://miro.com/app/board/uXjVOtmVf40=/
Data Pipelines & Services
  • EMIL - to add infra part...
  • Support the Structured Data teams implementation of SDAW grant work: Section Topics Data Pipeline (Q1) and Section Level Image Suggestions (Q2)
Emil (Infra)/Luke (Services) Need to consolidate or just use DE board
RESTBase Deprecation
  • Port API Backends used by Visual Editor to CORE
  • Migrate AQS 2.0 to k8s production
  • Review services to understand required changes
  • Investigate and understand access control needs
  • Define a RESTBase end of life plan and fully deprecate RESTBase
Luke Atieno #api_platform https://miro.com/app/board/uXjVOuAghYU=/
Shared Data Infrastructure Emil
Metrics Platform Emil
API Platform Desiree (Veronica)
Community Datasets (Dumps) Emil?
Platform Engineering Reliability
  • Upgrade Thumbor to Py3
  • Deploy Thumbor to Commons Beta and test
  • Deploy Thumbor to k8s
  • Tune Thumbor for prod traffic
  • Deprecate legacy Thumbor
Desiree Mat #thumbor https://miro.com/app/board/uXjVOtKCKBc=/


Rituals

  • 3 week sprints
  • Daily stand-ups (except Friday)
  • Retro after every sprint