Wikidata Platform/Airflow DAGs

Source code

On Gitlab: repos/data-engineering/airflow-dags

git clone git@gitlab.wikimedia.org:repos/data-engineering/airflow-dags.git

All DAGs are stored in this monorepo. The Wikidata Platform DAGs are stored in the wikidata subdirectory.

Deployment

DAG deployment is generally automatic within 5 minutes of a merge.

• Data_Platform/Systems/Airflow/Kubernetes#DAGs_deployment

Artifacts deployment is initiated manually after a successful merge.

• Data_Platform/Systems/Airflow/Kubernetes#Artifacts_deployment

Monitoring

• Web UI: https://airflow-wikidata.wikimedia.org/home

Other links

• Data_Platform/Systems/Airflow/Instances#wikidata