Help : Toolforge/Python

Python is one of the most used programming languages on Toolforge.

If you don't have a special python setup or want to use the latest versions, you should preferably use the recommended build service , for which there's several tutorials available.

For guidance on writing tools in Python, see e. g. Help:Toolforge/My first Flask OAuth tool , Help:Toolforge/My first Django OAuth tool and the build service based tutorials .

For this tutorial, we will be using prebuilt container images , the newest Python available on them is python3.13 .

You can not directly create a virtual environment on the bastion servers. You must first open a Kubernetes shell using


           toolforge webservice python3.13 shell

.

Virtual environments with prebuilt images

When using a prebuilt image (such as python3.13 ) with a webservice or a job , a Python virtual environment ( venv ) allows the maintainers of a tool to install Python packages using pip without affecting any other tools running in Toolforge. A venv is similar to the use of composer to install PHP libraries or bundler to install ruby libraries that are local to directory rather than system wide.

A virtual environment must be created in the same execution environment that the venv will be used from. This means that any virtual enviroments must be bootstrapped inside a container using toolforge webservice python3.13 shell or a job.

Web services

See Toolforge webservices with Python .

Jobs

Follow these instructions if you are using the Toolforge Jobs framework .

You will need to run a one-time manual job to bootstrap a python venv from inside a job container. This is similar to the instructions about using toolforge webservice python3.13 shell when bootstrapping kubernetes webservices.

Create a script similar to this to create the venv:

$HOME/bootstrap_venv.sh

#!/bin/bash

# use bash strict mode
set -euo pipefail

# delete the venv, if it already exists
rm -rf pyvenv

# create the venv
python3 -m venv pyvenv

# activate it
source pyvenv/bin/activate

# upgrade pip inside the venv and add support for the wheel package format
pip install -U pip wheel

# Change the following section depending on what your tool needs!

# install some concrete packages
# pip install requests
# pip install pyyaml

# or, install all packages from src/requirements.txt
# pip install -r src/requirements.txt

Run the bootstrapping script in the desired python container:

tools.mytool@tools-sgebastion-11:~$ ls bootstrap_venv.sh
bootstrap_venv.sh
tools.mytool@tools-sgebastion-11:~$ chmod ug+x bootstrap_venv.sh
tools.mytool@tools-sgebastion-11:~$ toolforge jobs run bootstrap-venv --command "cd $PWD && ./bootstrap_venv.sh" --image python3.13 --wait
tools.mytool@tools-sgebastion-11:~$ ls pyvenv
pyvenv

Once the venv has been created, you can run your python job as many times as needed as a manual, continuous, or periodic job:

tools.mytool@tools-sgebastion-11:~$ cat src/mytool.py
import requests
r = requests.get('https://www.wikidata.org/wiki/Q1')
print(r.status_code)
tools.mytool@tools-sgebastion-11:~$ toolforge jobs run mytool --command "pyvenv/bin/python src/mytool.py" --image python3.13
tools.mytool@tools-sgebastion-11:~$ cat mytool.out
200

Or, if you're using a shell script as the entry point for your job, use source pyvenv/bin/activate before running any other commands.

For scheduling recurring or continuous jobs, see Help:Toolforge/Jobs framework .

Elasticsearch

See Help:Toolforge/Elasticsearch#Python considerations .

Communication and support

Support and administration of the WMCS resources is provided by Wikimedia Foundation staff and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support

Chat in real time in the IRC channel #wikimedia-cloud ^connect or the bridged Telegram group
Discuss via email after you have subscribed to the cloud@ mailing list

Stay aware of critical changes and plans

Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
Read the News wiki page

Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read posts about Cloud Services on the Wikimedia Technical Blog . (Older posts can be read from the Cloud Services Blog .)

Virtual environments with prebuilt images

Web services

Jobs

Elasticsearch

See also

Communication and support