Help:Toolforge/Web/Python

Overview

This page describes Python-specific instructions for deploying a web server on Toolforge. Python web servers on Toolforege use uWSGI which is a Web Server Gateway Interface server for Python web applications. uWSGI can run applicaions built with Flask, Django, and other Python web application frameworks.

Starting a Python web service

To start a Python web service, use the webservice start command. Specify a backend and uwsgi configuration.

For backend, specify either kubernetes or gridengine. For uwsgi-configuration, specify either a Python version or uwsgi-plain for a custom configuration. For example:

• Python3.7 with a default uwsgi configuration:

webservice --backend=kubernetes python3.7 start

• Python3.5 with a default uwsgi configuration (deprecated):

webservice --backend=kubernetes python3.5 start

• Python3.4 with a default uwsgi configuration (deprecated):

webservice --backend=kubernetes python2 start

• Python2 with a default uwsgi configuration (deprecated)

webservice --backend=kubernetes python start

• Python2 on Grid Engine with a default uwsgi configuration

webservice --backend=gridengine uwsgi-python start

• Python2 or Python3 on Grid Engine with a user supplied uwsgi configuration

webservice --backend=gridengine uwsgi-plain start

uwsgi configuration

When you start a Python web service, can use a default uwsgi configuration or supply your own.

Default configuration

By default, Toolforge provides a common uWSGI configuration useful for a typical Python web service. This configuration uses a convention over configuration design with the following expectations:

• Your application uses a wsgi entry point in $HOME/www/python/src/app.py in a variable named app (example).
• Python libraries load from a virtualenv located in $HOME/www/python/venv.
• Custom configuration for uWSGI in ini file form will be loaded from $HOME/www/python/uwsgi.ini
  • Examples of configuration parameters can be found in the uWSGI manual.
  • Headers can be added using route = .* addheader:Access-Control-Allow-Origin: *
• Logs will be written to $HOME/uwsgi.log

Using a custom uwsgi configuration

To use a custom uwsgi configuation, specify uwsgi-plain:

webservice --backend=gridengine uwsgi-plain start|stop|restart

uwsgi-plain leaves configuration of the uWSGI service up to your tool's $HOME/uwsgi.ini configuration file. This allows you to tune the uWSGI service to work with your application.

To run a Python3 webservice on Grid Engine, you must use a custom uwsgi configuration. A working config for a Python3 Flask app on Grid Engine is documented in Phabricator task T104374.

Working with python3.7 (Python3 + Kubernetes)

This section includes notes and guides helpful when working with Python3 on Kubernetes: webservice --backend=kubernetes python3.7 start|stop|restart|shell

Virtualenv

Python3.7 runs with virtualenv support. You must use a virtualenv for installing your libraries.

Using virtualenv with webservice shell

You need to setup and use a new virtualenv. You can do so with the following:

For new projects

First, set up your python code so that your app.py file lives under ~/www/python/src. Then...

1. webservice --backend=kubernetes python3.7 shell
2. mkdir -p ~/www/python
3. python3 -m venv ~/www/python/venv (on a Toolforge bastion, use virtualenv -p python3 venv)
4. source ~/www/python/venv/bin/activate
5. pip install --upgrade pip wheel (This brings in newest pip, which is required for wheel support)
6. Install the libraries you need (e.g. pip install -r ~/www/python/src/requirements.txt)
7. exit out of webservice shell
8. webservice --backend=kubernetes python3.7 start

Step 1 can possibly freeze with an error message Pod is not ready in time. Retrying the command again should fix it.

Steps 2-6 can be automated by using the webservice-python-bootstrap script inside the webservice shell. If you want to create a brand new virtualenv in case you're switching Python versions or have new dependencies, pass --fresh.

Moving an existing project

If you are already running a Python3 Web service using uwsgi-plain on the job grid:

1. Make a backup of your current venv: mv ~/www/python/venv ~/www/python/venv.gridengine
2. Move your uwsgi.ini file away as well: mv ~/www/python/uwsgi.ini ~/www/python/uwsgi.ini.gridengine
3. Follow the instructions #For new projects
4. Before doing webservice --backend=kubernetes python start, you have to do a webservice --backend=gridengine stop
5. To switch back to gridengine, you can do:
   1. mv ~/www/python/venv ~/www/python/venv.k8s
   2. mv ~/www/python/venv.gridengine ~/www/python/venv
   3. mv ~/www/python/uwsgi.ini.gridengine ~/www/python/uwsgi.ini
   4. webservice --backend=kubernetes stop
   5. webservice --backend=gridengine uwsgi-plain start

The fundamental thing to remember is that virtualenvs created straight on the bastion work only with gridengine, and virtualenvs created inside webservice shell work only with kubernetes.

Once you are done migrating and are happy with it, you can delete your venv & uwsgi.ini backups.

Installing numpy / scipy / things with binary dependencies

If your package with binary dependencies has a manylinux1 wheel, you can directly install it with pip quickly and with minimum hassle. You can check if your package has a manylinux1 wheel by:

1. Go to https://pypi.python.org/pypi
2. Search for your package name in top right
3. Find it in the list and click on it
4. Look for packages that end in the string: cp34-cp34m-manylinux1_x86_64.whl
5. If it exists, then this package is installable with a binary wheel!

You can install it by:

1. webservice --backend=kubernetes python shell
2. source ~/www/python/venv/bin/activate
3. pip install --upgrade pip (This brings in newest pip, which is required for wheel support)
4. pip install $packagename

Tada! You only need to do the pip install --upgrade pip once, after that you can install manylinux1 packages easily.

Note that this only applies if you are using a package with binary dependencies. Most python packages do not have binary dependencies (are pure python) and do not need this!

Python/Python3.5 (Python3 + Kubernetes)

This works mostly like python3.7, but for Python 3.4 respectivly 3.5. These are outdated versions of Python that are no longer supported upstream and should not be used for new tools.

Python2 (Python2 + Kubernetes)

• webservice --backend=kubernetes python2 start|stop|restart|shell

See Default uwsgi configuration for general information.

uwsgi-python (Python2 + Grid Engine)

• webservice --backend=gridengine uwsgi-python start|stop|restart

See Default uwsgi configuration for general information. Python 3 is not supported by this type, but see the section on uwsgi-plain below for an alternative.

uwsgi-plain (Python3 + Grid Engine)

Tracked in Phabricator
Task T104374

• webservice --backend=gridengine uwsgi-plain start|stop|restart

The uwsgi-plain type leaves configuration of the uWSGI service up to the tool's $HOME/uwsgi.ini configuration file. This allows users with unique requirements to tune the uWSGI service to work with their application. One reason to use this is if you must run a Python3 webservice on Grid Engine. A working config for a Python3 Flask app is documented in Phabricator task T104374.

Using a uwsgi app with a default entry point that is not app.py

The default uwsgi configuration for the uwsgi webservice backend expects to find the uwsgi entry point as the variable app loaded from the $HOME/www/python/src/app.py module. If your application has another entry point, the easiest thing to do is create a $HOME/www/python/src/app.py module, import your entry point, and expose it as app. See Making a Django app work for an example of this pattern.

Making a Django app work

There is an issue that may currently need a workaround for Django: using utf8mb4 character and collation on your tables may cause issues with length of unique indexes, for instance when using python-social-auth or in your own models that have unique indexes. Using utf8 may cause errors when inserting 4-byte UTF-8 characters. See the issue for specific workarounds.

Setting up

By default your app.py should be in ~/www/python/src/. And contain:

import os

from django.core.wsgi import get_wsgi_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "<YOUR-TOOL-NAME>.settings")

app = get_wsgi_application()

To correctly locate the static files configure the place the uwsgi.ini into ~/www/python/uwsgi.ini. And add this setting:

[uwsgi]
check-static = /data/project/<YOUR-TOOL-NAME>/www/python

and in settings.py use:

STATIC_URL = '/<YOUR-TOOL-NAME>/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static')

Then deploy your static files into ~/www/python/static

Logs

You can find the logs in ~/uwsgi.log on both platforms.