Data Platform/Systems/Conda

Data Platform

Discover data

• Explore datasets in DataHub
• Data Lake
  • Traffic data
  • Edits data
  • Content data
  • Events data
• Analytics Query Service (AQS)

Access, query, and analyze data

• Get access to internal data
• Analytics tools
  • Jupyter notebooks
  • Superset
  • Spark
  • Presto
• Quickstart notebook
• Internal API requests

Transform and publish data

• Get help or file a request
• Plan data lifecyle
• Build tables and datasets
  • Dataset creation process
  • Data modeling guidelines
  • Airflow developer guide
  • Hive
  • Iceberg
  • Druid
• Share data and dashboards
  • Data publication guidelines
  • Turnilo
  • Superset
  • Web publication guide
  • Dashiki
• Manage published data
  • Data incident management
  • Data issue reporting
  • Data retention guidelines
  • Event data retention
  • Event sanitization
  • Dataset archiving and deletion

Collect data

• Test Kitchen
• Instrumentation tutorial
• Event Platform

---

Data Platform infrastructure and operations

• Systems overview
• Data pipelines
• Search
  • Using search for new features
  • Search Platform
  • Wikidata Query Service (WDQS)
• Operations and team processes
  • Ops week
  • Team pages on Wikitech
  • Team and project pages on MediaWiki.org

[ ]

We use Conda to manage packages and virtual environments on the stat hosts .

Environments are created by cloning Conda-Analytics, a custom Conda distribution maintained by the Data Platform Engineering team .

Use with Jupyter

For instructions on using Conda within our Jupyter environment, see Data Platform/Systems/Jupyter#Conda environments .

Use outside Jupyter

This section applies to Conda use outside of Jupyter (that is, when you connect to one of the analytics clients with a plain SSH terminal session).

In most cases, you can use the standard Conda commands (e.g. conda install , conda remove , conda list , conda deactivate ). This section covers the exceptions where we have custom commands to support our cloning-based workflow.

Creating a new environment

In the terminal, run conda-analytics-clone and a new clone of conda-analytics will be created for you in ~/.conda/envs .

It will be automatically named with the time and your username. If you prefer, you can give it a custom name: conda-analytics-clone my-cool-env .

Listing environments

$ conda-analytics-list
# conda environments:
#
2022-11-04T19.32.00_xcollazo     /home/xcollazo/.conda/envs/2022-11-04T19.32.00_xcollazo
2022-11-08T15.39.32_xcollazo     /home/xcollazo/.conda/envs/2022-11-08T15.39.32_xcollazo
2022-11-09T20.10.01_xcollazo     /home/xcollazo/.conda/envs/2022-11-09T20.10.01_xcollazo
base                  *  /opt/conda-analytics

Activating an environment

Run source conda-analytics-activate my-cool-env .

You can achieve the same thing with vanilla commands:

$ source /opt/conda-analytics/etc/profile.d/conda.sh
$ conda activate my-cool-env

You can also activate the read-only base environment, run source conda-analytics-activate base .

Installing packages

With a Conda environment activated, you can install packages by running conda install {{package}} in the terminal. If you are using Conda outside of Jupyter , you will first have to set your environment to use the HTTP proxy .

Conda will install packages from the Conda Forge channel by default. You can manually select a different channel by adding --channel {{channel}} to the command. The easiest way to search Conda Forge for a specific package is to do a regular web search with the qualifier "site:anaconda.org/conda-forge/".

If a Python package you need is not available from Conda Forge, you can use Pip instead.

Pinned package management

Each cloned environment comes with a pinned file whose main purpose is to prevent core packages from being automatically upgraded.

The pinned file is located in the conda-meta directory of each environment and can be customised to your liking

Troubleshooting

Spark 3 insert statement requirements

Using an INSERT statement in Spark 3 SQL or write.insertInto() in PySpark 3 results in the environment's Python executable being called. If the code is run from a cron job that loads a custom Python environment this might result in errors being thrown because that executable isn't available on the cluster. One way to solve this is to use wmfdata.spark.create_session(ship_python_env = True) to create a custom Spark session that ships the Python environment to the cluster nodes.

R support

R is not included by default, but can easily be installed. See Data Platform/Systems/R for details.

Administration

The Conda-Analytics base environment is based on Miniconda and has extra packages specific for our needs as well as scripts for cloning the environment. On the stat hosts , it is available in /opt/conda-analytics .

The code used to build new releases of conda-analytics lives in gitlab:repos/data-engineering/conda-analytics/ . The actual releases live in the associated package registry .