You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Help:Toolforge/Web/Python: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Xephyr826
 
imported>BryanDavis
Line 1: Line 1:
{{Template:Toolforge nav}}
== Overview ==
== Overview ==


This page describes Python-specific instructions for deploying a web server on [[Help:Toolforge/Web|Toolforge]]. Python web servers on Toolforege use [https://uwsgi-docs.readthedocs.io/en/latest/ uWSGI] which is a [[W:Web Server Gateway Interface|Web Server Gateway Interface]] server for Python web applications. uWSGI can run applicaions built with Flask, Django, and other Python web application frameworks.
This page describes Python-specific instructions for deploying a web server on [[Help:Toolforge/Web|Toolforge]]. Python web servers on Toolforege use [https://uwsgi-docs.readthedocs.io/en/latest/ uWSGI] which is a [[W:Web Server Gateway Interface|Web Server Gateway Interface]] (WSGI) server for Python web applications. uWSGI can run applicaions built with Flask, Django, and other Python web application frameworks.
 
== Conventions ==
 
The Toolforge <code>webservice</code> command starts Python applications using [[w:Convention over configuration|convention rather than configuration]]. These conventions are expected by the Toolforge tooling:
 
* Your WSGI application's entry point must be found in ''$HOME/www/python/src/app.py'' in a variable named <code>app</code> ([https://github.com/legoktm/fab-proxy/blob/08ec0de522cf0308e7c0687ee4f895d88696b3e4/app.py#L7 example]).
* Python libraries will be loaded from a virtual environment located in ''$HOME/www/python/venv''.
** On the Kubernetes backend, you '''must''' use a [[#venv|virtual environment]] to install and load any libraries you depend on outside of the Python standard library.
* Additional [https://uwsgi-docs.readthedocs.io/en/latest/Configuration.html#ini-files configuration for uWSGI] can be provided in a ''$HOME/www/python/uwsgi.ini'' file.
** Examples of configuration parameters can be found in the [https://uwsgi-docs.readthedocs.io/en/latest/Snippets.html uWSGI manual].
** Headers can be added using <code>route = .* addheader:Access-Control-Allow-Origin: *</code>
* Logs will be written to <code>$HOME/uwsgi.log</code>


== Starting a Python web service ==
== Starting a Python web service ==


To start a Python web service, use the [[Help:Toolforge/Web#Using_the_webservice_command|<code>webservice start</code>]] command. Specify a <code>backend</code> and <code>uwsgi configuration</code>.
To start a Python web service, use the [[Help:Toolforge/Web#Using_the_webservice_command|<code>webservice start</code>]] command. For example:


For <code>backend</code>, specify either [[Help:Toolforge/Web#Choosing_a_backend|<code>kubernetes</code> or <code>gridengine</code>]]. For <code>uwsgi-configuration</code>, specify either a Python version or <code>uwsgi-plain</code> for a custom configuration.  For example:
; Python3.7 with a default uwsgi configuration:
 
* Python3.7 with a default uwsgi configuration:
: <code>webservice --backend=kubernetes python3.7 start</code>
: <code>webservice --backend=kubernetes python3.7 start</code>
* Python3.5 with a default uwsgi configuration (deprecated):
; Python3.5 with a default uwsgi configuration (deprecated):
: <code>webservice --backend=kubernetes python3.5 start</code>
: <code>webservice --backend=kubernetes python3.5 start</code>
* Python3.4 with a default uwsgi configuration (deprecated):
; Python3.4 with a default uwsgi configuration (deprecated):
: <code>webservice --backend=kubernetes python start</code>
; Python2 with a default uwsgi configuration (deprecated)
: <code>webservice --backend=kubernetes python2 start</code>
: <code>webservice --backend=kubernetes python2 start</code>
* Python2 with a default uwsgi configuration (deprecated)
; Python2 on Grid Engine with a default uwsgi configuration
: <code>webservice --backend=kubernetes python start</code>
* Python2 on Grid Engine with a default uwsgi configuration
: <code>webservice --backend=gridengine uwsgi-python start</code>
: <code>webservice --backend=gridengine uwsgi-python start</code>
* Python2 or Python3 on Grid Engine with a user supplied uwsgi configuration
; Python2 or Python3 on Grid Engine with a user supplied uwsgi configuration
: <code>webservice --backend=gridengine uwsgi-plain start</code>
: <code>webservice --backend=gridengine uwsgi-plain start</code>


== uwsgi configuration ==
== {{Anchor|venv}}Virtual Environments and Packages ==
 
When you start a Python web service, can use a default uwsgi configuration or supply your own.
 
=== Default configuration ===
 
By default, Toolforge provides a common uWSGI configuration useful for a typical Python web service. This configuration uses a [[W:Convention over configuration|convention over configuration]] design with the following expectations:
 
* Your application uses a wsgi entry point in <code>$HOME/www/python/src/app.py</code> in a variable named <code>app</code> ([https://github.com/legoktm/fab-proxy/blob/08ec0de522cf0308e7c0687ee4f895d88696b3e4/app.py#L7 example]).
* Python libraries load from a virtualenv located in <code>$HOME/www/python/venv</code>.
* Custom configuration for uWSGI in [https://uwsgi-docs.readthedocs.io/en/latest/Configuration.html#ini-files ini file form] will be loaded from <code>$HOME/www/python/uwsgi.ini</code>
** Examples of configuration parameters can be found in the [https://uwsgi-docs.readthedocs.io/en/latest/Snippets.html?highlight=addheader uWSGI manual].
** Headers can be added using <code>route = .* addheader:Access-Control-Allow-Origin: *</code>
* Logs will be written to <code>$HOME/uwsgi.log</code>
 
=== Using a custom uwsgi configuration ===


To use a custom uwsgi configuation, specify <code>uwsgi-plain</code>:
A [https://docs.python.org/3/tutorial/venv.html virtual environment] ('''venv''') is a self-contained directory tree that contains a Python installation for a particular version of Python plus a number of additional packages. Using a venv allows you to install local Python packages for your tool.


<code>webservice --backend=gridengine uwsgi-plain start|stop|restart</code>
The fundamental thing to remember is that a venv created directly on the bastion will only work with ''--backend=gridengine'', and a venv created inside a webservice shell work only with ''--backend=kubernetes'' and the same Python runtime version.


<code>uwsgi-plain</code> leaves configuration of the uWSGI service up to your tool's <code>$HOME/uwsgi.ini</code> configuration file.
=== Creating a virtual environment ===
This allows you to tune the uWSGI service to work with your application.


To run a Python3 webservice on Grid Engine, you must use a custom uwsgi configuration. A working config for a Python3 Flask app on Grid Engine is documented in Phabricator task T104374.
# <code>webservice --backend=kubernetes python3.7 shell</code> (choose a different python version as appropriate for your project)
 
# <code>mkdir -p $HOME/www/python</code>
== Working with python3.7 (Python3 + Kubernetes) ==
# <code>python3 -m venv $HOME/www/python/venv</code>
This section includes notes and guides helpful when working with Python3 on Kubernetes:
# <code>source $HOME/www/python/venv/bin/activate</code>
<code>webservice --backend=kubernetes python3.7 start|stop|restart|shell</code>
 
=== Virtualenv ===
 
Python3.7 runs with virtualenv support. You '''must''' use a virtualenv for installing your libraries.
 
==== Using virtualenv with webservice shell ====
 
You need to setup and use a new virtualenv. You can do so with the following:
 
===== For new projects =====
First, set up your python code so that your <code>app.py</code> file lives under [https://github.com/physikerwelt/python-minimal ~/www/python/src]. Then...
 
# <code>webservice --backend=kubernetes python3.7 shell</code>
# <code>mkdir -p ~/www/python</code>
# <code>python3 -m venv ~/www/python/venv</code> (on a Toolforge bastion, use <code>virtualenv -p python3 venv</code>)
# <code>source ~/www/python/venv/bin/activate</code>
# <code>pip install --upgrade pip wheel</code> (This brings in newest pip, which is required for wheel support)
# <code>pip install --upgrade pip wheel</code> (This brings in newest pip, which is required for wheel support)
# Install the libraries you need (e.g. <code>pip install -r ~/www/python/src/requirements.txt</code>)
# Install the libraries you need (for example <code>pip install -r $HOME/www/python/src/requirements.txt</code>)
# exit out of webservice shell
# exit out of webservice shell
# <code>webservice --backend=kubernetes python3.7 start</code>
# <code>webservice --backend=kubernetes python3.7 start</code>
Line 74: Line 53:
Step 1 can possibly freeze with an error message <code>Pod is not ready in time</code>. Retrying the command again should fix it.  
Step 1 can possibly freeze with an error message <code>Pod is not ready in time</code>. Retrying the command again should fix it.  


Steps 2-6 can be automated by using the <code>webservice-python-bootstrap</code> script inside the webservice shell. If you want to create a brand new virtualenv in case you're switching Python versions or have new dependencies, pass <code>--fresh</code>.
Steps 2-6 can be automated by using the <code>webservice-python-bootstrap</code> script inside the webservice shell. If you want to create a brand new virtualenv in case you're switching Python versions or have new dependencies, use <code>webservice-python-bootstrap --fresh</code>.
 
===== Moving an existing project =====
 
If you are already running a Python3 Web service using uwsgi-plain on the job grid:
 
# Make a backup of your current venv: <code>mv ~/www/python/venv ~/www/python/venv.gridengine</code>
# Move your uwsgi.ini file away as well: <code>mv ~/www/python/uwsgi.ini ~/www/python/uwsgi.ini.gridengine</code>
# Follow the instructions [[#For new projects]]
# Before doing <code>webservice --backend=kubernetes python start</code>, you have to do a <code>webservice --backend=gridengine stop</code>
# To switch back to gridengine, you can do:
## <code>mv ~/www/python/venv ~/www/python/venv.k8s</code>
## <code>mv ~/www/python/venv.gridengine ~/www/python/venv</code>
## <code>mv ~/www/python/uwsgi.ini.gridengine ~/www/python/uwsgi.ini</code>
## <code>webservice --backend=kubernetes stop</code>
## <code>webservice --backend=gridengine uwsgi-plain start</code>
 
The fundamental thing to remember is that virtualenvs created straight on the bastion work only with gridengine, and virtualenvs created inside webservice shell work only with kubernetes.
 
Once you are done migrating and are happy with it, you can delete your venv & uwsgi.ini backups.
 
=== Installing numpy / scipy / things with binary dependencies ===
 
If your package with binary dependencies has a [https://github.com/pypa/manylinux manylinux1 wheel], you can directly install it with pip quickly and with minimum hassle. You can check if your package has a manylinux1 wheel by:
 
# Go to https://pypi.python.org/pypi
# Search for your package name in top right
# Find it in the list and click on it
# Look for packages that end in the string: <code>cp34-cp34m-manylinux1_x86_64.whl</code>
# If it exists, then this package is installable with a binary wheel!
 
You can install it by:
 
# <code>webservice --backend=kubernetes python shell</code>
# <code>source ~/www/python/venv/bin/activate</code>
# <code>pip install --upgrade pip</code> (This brings in newest pip, which is required for wheel support)
# <code>pip install $packagename</code>
 
Tada! You only need to do the <code>pip install --upgrade pip</code> once, after that you can install manylinux1 packages easily.
 
Note that this only applies if you are using a package with binary dependencies. Most python packages do not have binary dependencies (are pure python) and do not need this!
 
== Python/Python3.5 (Python3 + Kubernetes) ==
This works mostly like [[Help:Toolforge/Web#python3.7 (Python3 + Kubernetes)|python3.7]], but for Python 3.4 respectivly 3.5. These are outdated versions of Python that are no longer supported upstream and should not be used for new tools.
 
== Python2 (Python2 + Kubernetes) ==
 
* <code>webservice --backend=kubernetes python2 start|stop|restart|shell</code>
 
See [[#Default uwsgi configuration|Default uwsgi configuration]] for general information.
 
== uwsgi-python (Python2 + Grid Engine) ==
* <code>webservice --backend=gridengine uwsgi-python start|stop|restart</code>
 
See [[#Default uwsgi configuration|Default uwsgi configuration]] for general information. Python 3 is not supported by this type, but see the section on uwsgi-plain below for an alternative.
 
== uwsgi-plain (Python3 + Grid Engine) ==
{{Tracked|T104374}}


* <code>webservice --backend=gridengine uwsgi-plain start|stop|restart</code>
== Using a uwsgi app with a default entry point that is not app.py ==


The <code>uwsgi-plain</code> type leaves configuration of the uWSGI service up to the tool's <code>$HOME/uwsgi.ini</code> configuration file. This allows users with unique requirements to tune the uWSGI service to work with their application. One reason to use this is if you must run a Python3 webservice on Grid Engine. A working config for a Python3 Flask app is [[Phab:T104374#1911373|documented in Phabricator task T104374]].
The default uwsgi configuration for the uwsgi webservice backend expects to find the uwsgi entry point as the variable <code>app</code> loaded from the <code>$HOME/www/python/src/app.py</code> module. If your application has another entry point, the easiest thing to do is create a <code>$HOME/www/python/src/app.py</code> module, import your entry point, and expose it as <code>app</code>. See [[#Deploying a Django application|Deploying a Django application]] for an example of this pattern.


== Using a uwsgi app with a default entry point that is not <code>app.py</code> ==
== {{anchor|Making a Django app work}}Deploying a Django application ==
[[w:Django (web framework)|Django]] is a popular web framework for developing Python applications. A typical Django application will need a few changes to run with Toolforge's opinionated uWSGI configuration.


The default uwsgi configuration for the uwsgi webservice backend expects to find the uwsgi entry point as the variable <code>app</code> loaded from the <code>$HOME/www/python/src/app.py</code> module. If your application has another entry point, the easiest thing to do is create a <code>$HOME/www/python/src/app.py</code> module, import your entry point, and expose it as <code>app</code>. See [[#Making a Django app work|Making a Django app work]] for an example of this pattern.
=== Create an app.py entry point ===
 
{{Codesample|name=$HOME/www/python/src/app.py|lang=python|scheme=light|code=
== Making a Django app work ==
 
There is an issue that may currently need a workaround for Django: using <code>utf8mb4</code> character and collation on your tables may cause issues with length of unique indexes, for instance when using python-social-auth or in your own models that have unique indexes. Using <code>utf8</code> may cause errors when inserting 4-byte UTF-8 characters. See [https://phabricator.wikimedia.org/T198508 the issue] for specific workarounds.
 
==== Setting up ====
By default your <code>app.py</code> should be in <code>~/www/python/src/</code>. And contain:
 
<source lang="python">
import os
import os


Line 156: Line 71:


app = get_wsgi_application()
app = get_wsgi_application()
</source>
}}


To correctly locate the static files configure the place the <code>uwsgi.ini</code> into <code>~/www/python/uwsgi.ini</code>. And add this setting:
=== Static files ===
To correctly locate the static files, first configure ''$HOME/www/python/uwsgi.ini'' to look for static files in your tool's ''$HOME/www/python/src/static'' directory:


<pre>
{{Codesample|name=$HOME/www/python/uwsgi.ini|lang=ini|scheme=light|code=
[uwsgi]
[uwsgi]
check-static = /data/project/<YOUR-TOOL-NAME>/www/python
check-static = /data/project/<YOUR-TOOL-NAME>/www/python/src/static
</pre>
}}


and in <code>settings.py</code> use:
Next configure your Django app to use this location by editing the app's ''settings.py'' file:


<source lang="python">
{{Codesample|name=settings.py|lang=python|scheme=light|code=
STATIC_URL = '/<YOUR-TOOL-NAME>/static/'
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static')
STATIC_ROOT = os.path.join(BASE_DIR, 'static')
</source>
}}


Then deploy your static files into <code>~/www/python/static</code>
Finally deploy your static files into ''$HOME/www/python/src/static''. Typically this will be done by running <code>python manage.py collectstatic</code>.
 
=== MySQL and utf8mb4 ===
{{Tracked|T198508}}
 
The version of MySQL/MariaDB currently used by ToolsDB will not work transparently with Django and the [https://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html utf8mb4] character set. <code>ROW_FORMAT=DYNAMIC</code> must be used on all tables which will index utf8mb4 encoded character fields longer than 191 characters. This configuration, together with database server configuration to enable ''innodb_large_prefix'', the Barracuda file format, and file per table storage, enables index key prefixes longer than 767 bytes (up to 3072 bytes) for InnoDB tables.
 
Django does not have a feature flag or setting for adding the needed <code>ROW_FORMAT=DYNAMIC</code> configuration to its database migrations. One way to work around this issue is by using a [[gerrit:436592|custom database engine]] as {{U|Volans}} did for his Debmonitor project. Another possible workaround is manually modifying your migration files as {{U|BryanDavis}} [https://bd808.com/blog/2017/04/17/making-django-migrations-that-work-with-mysql-55-and-utf8mb4/ documented in a blog post].


== Logs ==
== Logs ==


You can find the logs in ~/uwsgi.log on both platforms.
You can find your application's log messages in ''$HOME/uwsgi.log''.
 
{{:Help:Cloud Services communication}}
 
==See also==
* [[Help:Toolforge/Web]]
* [[Help:Toolforge/Python]]
* [[Help:Toolforge/How to#Python]]
 
[[Category:Toolforge]]
[[Category:Documentation]]

Revision as of 00:36, 11 August 2020

Overview

This page describes Python-specific instructions for deploying a web server on Toolforge. Python web servers on Toolforege use uWSGI which is a Web Server Gateway Interface (WSGI) server for Python web applications. uWSGI can run applicaions built with Flask, Django, and other Python web application frameworks.

Conventions

The Toolforge webservice command starts Python applications using convention rather than configuration. These conventions are expected by the Toolforge tooling:

  • Your WSGI application's entry point must be found in $HOME/www/python/src/app.py in a variable named app (example).
  • Python libraries will be loaded from a virtual environment located in $HOME/www/python/venv.
    • On the Kubernetes backend, you must use a virtual environment to install and load any libraries you depend on outside of the Python standard library.
  • Additional configuration for uWSGI can be provided in a $HOME/www/python/uwsgi.ini file.
    • Examples of configuration parameters can be found in the uWSGI manual.
    • Headers can be added using route = .* addheader:Access-Control-Allow-Origin: *
  • Logs will be written to $HOME/uwsgi.log

Starting a Python web service

To start a Python web service, use the webservice start command. For example:

Python3.7 with a default uwsgi configuration
webservice --backend=kubernetes python3.7 start
Python3.5 with a default uwsgi configuration (deprecated)
webservice --backend=kubernetes python3.5 start
Python3.4 with a default uwsgi configuration (deprecated)
webservice --backend=kubernetes python start
Python2 with a default uwsgi configuration (deprecated)
webservice --backend=kubernetes python2 start
Python2 on Grid Engine with a default uwsgi configuration
webservice --backend=gridengine uwsgi-python start
Python2 or Python3 on Grid Engine with a user supplied uwsgi configuration
webservice --backend=gridengine uwsgi-plain start

Virtual Environments and Packages

A virtual environment (venv) is a self-contained directory tree that contains a Python installation for a particular version of Python plus a number of additional packages. Using a venv allows you to install local Python packages for your tool.

The fundamental thing to remember is that a venv created directly on the bastion will only work with --backend=gridengine, and a venv created inside a webservice shell work only with --backend=kubernetes and the same Python runtime version.

Creating a virtual environment

  1. webservice --backend=kubernetes python3.7 shell (choose a different python version as appropriate for your project)
  2. mkdir -p $HOME/www/python
  3. python3 -m venv $HOME/www/python/venv
  4. source $HOME/www/python/venv/bin/activate
  5. pip install --upgrade pip wheel (This brings in newest pip, which is required for wheel support)
  6. Install the libraries you need (for example pip install -r $HOME/www/python/src/requirements.txt)
  7. exit out of webservice shell
  8. webservice --backend=kubernetes python3.7 start

Step 1 can possibly freeze with an error message Pod is not ready in time. Retrying the command again should fix it.

Steps 2-6 can be automated by using the webservice-python-bootstrap script inside the webservice shell. If you want to create a brand new virtualenv in case you're switching Python versions or have new dependencies, use webservice-python-bootstrap --fresh.

Using a uwsgi app with a default entry point that is not app.py

The default uwsgi configuration for the uwsgi webservice backend expects to find the uwsgi entry point as the variable app loaded from the $HOME/www/python/src/app.py module. If your application has another entry point, the easiest thing to do is create a $HOME/www/python/src/app.py module, import your entry point, and expose it as app. See Deploying a Django application for an example of this pattern.

Deploying a Django application

Django is a popular web framework for developing Python applications. A typical Django application will need a few changes to run with Toolforge's opinionated uWSGI configuration.

Create an app.py entry point

$HOME/www/python/src/app.py
import os

from django.core.wsgi import get_wsgi_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "<YOUR-TOOL-NAME>.settings")

app = get_wsgi_application()

Static files

To correctly locate the static files, first configure $HOME/www/python/uwsgi.ini to look for static files in your tool's $HOME/www/python/src/static directory:

$HOME/www/python/uwsgi.ini
[uwsgi]
check-static = /data/project/<YOUR-TOOL-NAME>/www/python/src/static

Next configure your Django app to use this location by editing the app's settings.py file:

settings.py
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static')

Finally deploy your static files into $HOME/www/python/src/static. Typically this will be done by running python manage.py collectstatic.

MySQL and utf8mb4

Tracked in Phabricator
Task T198508

The version of MySQL/MariaDB currently used by ToolsDB will not work transparently with Django and the utf8mb4 character set. ROW_FORMAT=DYNAMIC must be used on all tables which will index utf8mb4 encoded character fields longer than 191 characters. This configuration, together with database server configuration to enable innodb_large_prefix, the Barracuda file format, and file per table storage, enables index key prefixes longer than 767 bytes (up to 3072 bytes) for InnoDB tables.

Django does not have a feature flag or setting for adding the needed ROW_FORMAT=DYNAMIC configuration to its database migrations. One way to work around this issue is by using a custom database engine as Volans did for his Debmonitor project. Another possible workaround is manually modifying your migration files as BryanDavis documented in a blog post.

Logs

You can find your application's log messages in $HOME/uwsgi.log.

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
Receive mail announcements about critical changes
Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
Track work tasks and report bugs
Use the Phabricator workboard #Cloud-Services for bug reports and feature requests about the Cloud VPS infrastructure itself
Learn about major near-term plans
Read the News wiki page
Read news and stories about Wikimedia Cloud Services
Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)

See also

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Help:Toolforge/Web/Python&oldid=537130"