You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Help:Toolforge/My first Flask OAuth tool: Difference between revisions
imported>Lucas Werkmeister (WMDE) m (→Next Steps: remove “enforce HTTPS” from toolforge library description, now enforced at the Toolforge level) |
imported>David Caro (Added notes pointing to oauth v1 only) |
||
(15 intermediate revisions by 9 users not shown) | |||
Line 1: | Line 1: | ||
{{Toolforge nav}} | {{Toolforge nav}} | ||
== Overview == | == Overview == | ||
Line 12: | Line 10: | ||
* Create a new tool | * Create a new tool | ||
* Run a Python 3 [[w:Web Server Gateway Interface|WSGI]] webservice on [[Help:Toolforge/Kubernetes|Kubernetes]] | * Run a Python 3 [[w:Web Server Gateway Interface|WSGI]] webservice on [[Help:Toolforge/Kubernetes|Kubernetes]] | ||
* Allow webservice visitors to authenticate via [[Mw:Help:OAuth|OAuth]] using their [[W:Wikipedia:Unified login|Wikimedia unified account]] | * Allow webservice visitors to authenticate via [[Mw:Help:OAuth|OAuth]] v1 using their [[W:Wikipedia:Unified login|Wikimedia unified account]] | ||
== Getting started == | == Getting started == | ||
Line 23: | Line 21: | ||
* Basic knowledge of [[W:Secure Shell|SSH]] | * Basic knowledge of [[W:Secure Shell|SSH]] | ||
* Basic knowledge of the [[W:Unix|Unix command line]] | * Basic knowledge of the [[W:Unix|Unix command line]] | ||
* Familiarity with [[Mw:OAuth/For Developers|OAuth]] terminology would be nice but not required | * Familiarity with [[Mw:OAuth/For Developers|OAuth]] (v1) terminology would be nice but not required | ||
==== Accounts ==== | ==== Accounts ==== | ||
Line 29: | Line 27: | ||
* [[Help:Getting Started|A Toolforge account]] | * [[Help:Getting Started|A Toolforge account]] | ||
== | == Step-by-step guide == | ||
* Create a new tool account | * Step 1: Create a new tool account | ||
* Create a basic Flask WSGI webservice | * Step 2: Create a basic Flask WSGI webservice | ||
* Add a configuration file | * Step 3: Add a configuration file | ||
* Add support for | * Step 4: Add support for OAuth1 authentication | ||
== Step | === Step 1: Create a new tool account === | ||
# Create [[Portal:Toolforge/Tool Accounts#Create tools with Tool Accounts|a new tool account]]. | |||
# Create [[ | |||
#* For the examples in this tutorial, <code><TOOL NAME></code> is used to indicate places where your unique tool name is used in another command. | #* For the examples in this tutorial, <code><TOOL NAME></code> is used to indicate places where your unique tool name is used in another command. | ||
# SSH to login. | # SSH to login.toolforge.org. | ||
#* If your local username is different from your shell account name on Toolforge, you will need to include your Toolforge shell account name in your ssh command (i.e. <shell_username>@login. | #* If your local username is different from your shell account name on Toolforge, you will need to include your Toolforge shell account name in your ssh command (i.e. <shell_username>@login.toolforge.org). | ||
# Run <code>become <TOOL NAME></code> to change to the tool user. | # Run <code>become <TOOL NAME></code> to change to the tool user. | ||
== Step 2: Create a basic Flask WSGI webservice == | === Step 2: Create a basic Flask WSGI webservice === | ||
;What is Flask? | |||
[[W:Flask (web framework)|Flask]] is a popular web development framework for Python. | [[W:Flask (web framework)|Flask]] is a popular web development framework for Python. | ||
=== How to create a basic Flask WSGI webservice === | ==== How to create a basic Flask WSGI webservice ==== | ||
;Create the $HOME/www/python/src directory for your application: | |||
{{Codesample|lang=shell-session|scheme=light|code= | |||
$ mkdir -p $HOME/www/python/src | $ mkdir -p $HOME/www/python/src | ||
}} | |||
;Create a Python virtual environment for the application's external library dependencies | |||
The virtual environment allows the tool to install Python libraries locally without needing a Toolforge administrator's help. The default <code>webservice</code> configuration will automatically load libraries from <code>$HOME/www/python/venv</code>. | The virtual environment allows the tool to install Python libraries locally without needing a Toolforge administrator's help. The default <code>webservice</code> configuration will automatically load libraries from <code>$HOME/www/python/venv</code>. | ||
Line 66: | Line 64: | ||
This will ensure that the version of Python the virtual environment uses matches the version of Python used by the Kubernetes runtime. | This will ensure that the version of Python the virtual environment uses matches the version of Python used by the Kubernetes runtime. | ||
{{Codesample|lang=shell-session|scheme=light|code= | |||
$ webservice --backend=kubernetes | $ webservice --backend=kubernetes python3.7 shell | ||
If you don't see a command prompt, try pressing enter. | If you don't see a command prompt, try pressing enter. | ||
$ python3 -m venv $HOME/www/python/venv | $ python3 -m venv $HOME/www/python/venv | ||
Line 76: | Line 74: | ||
Successfully installed pip | Successfully installed pip | ||
Cleaning up... | Cleaning up... | ||
}} | |||
If you run <code>python3 -m venv $HOME/www/python/venv</code> and get an error beginning with "The virtual environment was not created successfully because ensurepip is not available.", you didn't run the first command (the "webservice" one) first. | If you run <code>python3 -m venv $HOME/www/python/venv</code> and get an error beginning with "The virtual environment was not created successfully because ensurepip is not available.", you didn't run the first command (the "webservice" one) first. | ||
;Add Flask to the virtual environment | |||
'''Note:''' It is Python best practice to use a file named <code>requirements.txt</code> to keep track of the library dependencies of applications. | '''Note:''' It is Python best practice to use a file named <code>requirements.txt</code> to keep track of the library dependencies of applications. | ||
{{Codesample|lang=shell-session|scheme=light|code= | |||
$ cat > $HOME/www/python/src/requirements.txt << EOF | $ cat > $HOME/www/python/src/requirements.txt << EOF | ||
flask | flask | ||
Line 92: | Line 90: | ||
[...] | [...] | ||
Successfully installed [...] | Successfully installed [...] | ||
}} | |||
The initial virtual environment is now set-up. Exit out of the Kubernetes shell and return to the SSH session on the bastion. | The initial virtual environment is now set-up. Exit out of the Kubernetes shell and return to the SSH session on the bastion. | ||
{{Codesample|lang=shell-session|scheme=light|code= | |||
$ exit | $ exit | ||
}} | |||
;Create a 'hello world' WSGI application | |||
{{Codesample|lang=shell-session|scheme=light|name=$HOME/www/python/src/app.py|code= | |||
# -*- coding: utf-8 -*- | # -*- coding: utf-8 -*- | ||
# | # | ||
Line 133: | Line 129: | ||
def index(): | def index(): | ||
return 'Hello World!' | return 'Hello World!' | ||
}} | |||
'''Note:''' The 'Hello World!' file above starts with a license header that places it under the [https://www.gnu.org/copyleft/gpl.html GPLv3+ license]. | '''Note:''' The 'Hello World!' file above starts with a license header that places it under the [https://www.gnu.org/copyleft/gpl.html GPLv3+ license]. | ||
Line 139: | Line 135: | ||
Code on Toolforge should always be licensed under an [https://opensource.org/licenses Open Source Initiative (OSI) approved license]. See the [[Help:Toolforge/Right to fork policy|Right to fork policy]] for more information on this Toolforge policy. | Code on Toolforge should always be licensed under an [https://opensource.org/licenses Open Source Initiative (OSI) approved license]. See the [[Help:Toolforge/Right to fork policy|Right to fork policy]] for more information on this Toolforge policy. | ||
;Start the webservice | |||
$ webservice --backend=kubernetes | {{Codesample|lang=shell-session|scheme=light|code= | ||
$ webservice --backend=kubernetes python3.7 start | |||
Starting webservice. | Starting webservice. | ||
}} | |||
Once the webservice is started, navigate to <code><nowiki>https:/ | Once the webservice is started, navigate to <code><nowiki>https://</nowiki><TOOL NAME>.toolforge.org/</code> in your web browser, and see a 'Hello World!' message. It might take a few minutes until it is reachable. | ||
=== Notes === | ==== Notes ==== | ||
Toolforge uses a uWSGI configuration designed to make it easy to deploy a typical Python webservice. This configuration uses a '[[w:Convention over configuration|convention over configuration]]' design with the following expectations: | Toolforge uses a uWSGI configuration designed to make it easy to deploy a typical Python webservice. This configuration uses a '[[w:Convention over configuration|convention over configuration]]' design with the following expectations: | ||
* The application will have a WSGI entry point in <code>$HOME/www/python/src/app.py</code> in a variable named <code>app</code>. | * The application will have a WSGI entry point in <code>$HOME/www/python/src/app.py</code> in a variable named <code>app</code>. | ||
Line 152: | Line 149: | ||
* Logs will be written to <code>$HOME/uwsgi.log</code> | * Logs will be written to <code>$HOME/uwsgi.log</code> | ||
;Expected file layout | |||
{{Codesample|lang=text|scheme=light|code= | |||
$HOME | $HOME | ||
├─ uwsgi.log | ├─ uwsgi.log | ||
Line 161: | Line 158: | ||
│ └─ app.py | │ └─ app.py | ||
└─ venv | └─ venv | ||
}} | |||
=== Troubleshooting === | ==== Troubleshooting ==== | ||
If there is an error when the webservice is started, look in <code>$HOME/uwsgi.log</code> and <code>$HOME/error.log</code> for an explanation. | If there is an error when the webservice is started, look in <code>$HOME/uwsgi.log</code> and <code>$HOME/error.log</code> for an explanation. | ||
Line 169: | Line 166: | ||
One Unix utility to use for this is <code>[https://linux.die.net/man/1/tail tail]</code>, which will display lines from the end of a file: | One Unix utility to use for this is <code>[https://linux.die.net/man/1/tail tail]</code>, which will display lines from the end of a file: | ||
{{Codesample|lang=shell-session|scheme=light|code= | |||
$ tail -n 50 $HOME/uwsgi.log | $ tail -n 50 $HOME/uwsgi.log | ||
$ tail -n 50 $HOME/error.log | $ tail -n 50 $HOME/error.log | ||
}} | |||
== Step 3: Add a configuration file == | === Step 3: Add a configuration file === | ||
The application will eventually need some configuration data like OAuth secrets or passwords. These should not be hard coded into the Python files, because the secrets and passwords will be visible once the source code is made public. | The application will eventually need some configuration data like OAuth secrets or passwords. These should not be hard coded into the Python files, because the secrets and passwords will be visible once the source code is made public. | ||
Line 180: | Line 177: | ||
There are many different ways to separate code from configuration, but the most straight forward when using Flask is to keep the configuration in a file that can be parsed easily, and then add it to the <code>app.config</code> object that Flask provides. | There are many different ways to separate code from configuration, but the most straight forward when using Flask is to keep the configuration in a file that can be parsed easily, and then add it to the <code>app.config</code> object that Flask provides. | ||
=== How to add a configuration file === | ==== How to add a configuration file ==== | ||
;Add PyYAML to the virtual environment | |||
In this tutorial, a YAML file is used to hold secrets. YAML is a nice choice, because it has a simple syntax, is fairly easy for humans to read, and supports both comments and complex values like lists and dictionaries. | In this tutorial, a YAML file is used to hold secrets. YAML is a nice choice, because it has a simple syntax, is fairly easy for humans to read, and supports both comments and complex values like lists and dictionaries. | ||
Line 188: | Line 185: | ||
Python does not have built in support for parsing YAML files. Install a library to help out. | Python does not have built in support for parsing YAML files. Install a library to help out. | ||
{{Codesample|lang=shell-session|scheme=light|code= | |||
$ webservice --backend=kubernetes | $ webservice --backend=kubernetes python3.7 shell | ||
If you don't see a command prompt, try pressing enter. | If you don't see a command prompt, try pressing enter. | ||
$ source $HOME/www/python/venv/bin/activate | $ source $HOME/www/python/venv/bin/activate | ||
Line 201: | Line 198: | ||
Successfully installed pyyaml | Successfully installed pyyaml | ||
$ exit | $ exit | ||
}} | |||
;Read configuration from a file | |||
Update the <code>$HOME/www/python/src/app.py</code> file to read configuration from a <code>config.yaml</code> file in the same directory and get the greeting from the configuration file: | Update the <code>$HOME/www/python/src/app.py</code> file to read configuration from a <code>config.yaml</code> file in the same directory and get the greeting from the configuration file: | ||
{{Codesample|lang=python3|scheme=light|name=$HOME/www/python/src/app.py|code= | |||
# -*- coding: utf-8 -*- | # -*- coding: utf-8 -*- | ||
# | # | ||
Line 244: | Line 241: | ||
def index(): | def index(): | ||
return app.config['GREETING'] | return app.config['GREETING'] | ||
}} | |||
A configuration file is now required or the application will produce an error. Eventually, secrets will be included in this file. The file's permissions should be changed so that only the tool user can read it. | A configuration file is now required or the application will produce an error. Eventually, secrets will be included in this file. The file's permissions should be changed so that only the tool user can read it. | ||
{{Codesample|lang=shell-session|scheme=light|code= | |||
$ touch $HOME/www/python/src/config.yaml | $ touch $HOME/www/python/src/config.yaml | ||
$ chmod u=rw,go= $HOME/www/python/src/config.yaml | $ chmod u=rw,go= $HOME/www/python/src/config.yaml | ||
Line 254: | Line 251: | ||
GREETING: Goodnight moon! | GREETING: Goodnight moon! | ||
EOF | EOF | ||
}} | |||
Now restart the webservice: | |||
{{Codesample|lang=shell-session|scheme=light|code= | |||
$ webservice restart | $ webservice restart | ||
Restarting webservice... | Restarting webservice... | ||
}} | |||
Once the webservice has restarted, navigate to <code><nowiki>https:/ | Once the webservice has restarted, navigate to <code><nowiki>https://</nowiki><TOOL NAME>.toolforge.org/</code> in your web browser and see the new 'Goodnight moon!' message. | ||
==== Troubleshooting ==== | ==== Troubleshooting ==== | ||
Line 265: | Line 264: | ||
If there is an error, look in <code>$HOME/uwsgi.log</code> and <code>$HOME/error.log</code> for an explanation. | If there is an error, look in <code>$HOME/uwsgi.log</code> and <code>$HOME/error.log</code> for an explanation. | ||
== Step 4: Add support for OAuth authentication == | === Step 4: Add support for OAuth authentication === | ||
OAuth is a safe mechanism for authenticating a Wikimedia user in the application. For the basics, [[Mw:OAuth/For Developers|read more about OAuth on mediawiki.org]]. | OAuth is a safe mechanism for authenticating a Wikimedia user in the application. For the basics, [[Mw:OAuth/For Developers|read more about OAuth on mediawiki.org]]. | ||
=== How to add mwoauth to the virtual environment === | ==== How to add mwoauth to the virtual environment ==== | ||
[https://pythonhosted.org/mwoauth/ mwoauth library] is used to handle most of the complexity of making OAuth requests to MediaWiki. | [https://pythonhosted.org/mwoauth/ mwoauth library] is used to handle most of the complexity of making OAuth requests to MediaWiki. | ||
{{Codesample|lang=shell-session|scheme=light|code= | |||
$ webservice --backend=kubernetes | $ webservice --backend=kubernetes python3.7 shell | ||
If you don't see a command prompt, try pressing enter. | If you don't see a command prompt, try pressing enter. | ||
$ source $HOME/www/python/venv/bin/activate | $ source $HOME/www/python/venv/bin/activate | ||
Line 287: | Line 286: | ||
Successfully installed [...] | Successfully installed [...] | ||
$ exit | $ exit | ||
}} | |||
;Update the application code | |||
Here is our new <code>$HOME/www/python/src/app.py</code> file: | Here is our new <code>$HOME/www/python/src/app.py</code> file: | ||
{{Collapse top|www/python/src/app.py}} | {{Collapse top|www/python/src/app.py}} | ||
{{Codesample|lang=python3|scheme=light|name=$HOME/www/python/src/app.py|code= | |||
# -*- coding: utf-8 -*- | # -*- coding: utf-8 -*- | ||
# | # | ||
Line 391: | Line 390: | ||
flask.session.clear() | flask.session.clear() | ||
return flask.redirect(flask.url_for('index')) | return flask.redirect(flask.url_for('index')) | ||
}} | |||
{{Collapse bottom}} | {{Collapse bottom}} | ||
Line 397: | Line 396: | ||
By default Flask will look for templates in the <code>$HOME/www/python/src/templates</code> directory. | By default Flask will look for templates in the <code>$HOME/www/python/src/templates</code> directory. | ||
{{Codesample|lang=shell-session|scheme=light|code= | |||
$ mkdir $HOME/www/python/src/templates | $ mkdir $HOME/www/python/src/templates | ||
$ edit $HOME/www/python/src/templates/index.html | $ edit $HOME/www/python/src/templates/index.html | ||
}} | |||
{{Codesample|lang=html+jinja|scheme=light|name=$HOME/www/python/src/templates/index.html|code= | |||
<!DOCTYPE HTML> | <!DOCTYPE HTML> | ||
<html> | <html> | ||
Line 408: | Line 408: | ||
<body> | <body> | ||
{% if username %} | {% if username %} | ||
<p>Hello {{ username }}!</p> | <p>Hello {{((}} username {{))}}!</p> | ||
<p><a href="{{ url_for('logout') }}">logout</a></p> | <p><a href="{{((}} url_for('logout') {{))}}">logout</a></p> | ||
{% else %} | {% else %} | ||
<p>{{ greeting }}</p> | <p>{{((}} greeting {{))}}</p> | ||
<p><a href="{{ url_for('login') }}">login</a></p> | <p><a href="{{((}} url_for('login') {{))}}">login</a></p> | ||
{% endif %} | {% endif %} | ||
</body> | </body> | ||
</html> | </html> | ||
}} | |||
;Update the configuration to add OAuth secrets | |||
Add new configuration values to <code>$HOME/www/python/src/config.yaml</code> file to go with the new code. | |||
# [https://meta.wikimedia.org/wiki/Special:OAuthConsumerRegistration/propose Register a new OAuth consumer]. | # [https://meta.wikimedia.org/wiki/Special:OAuthConsumerRegistration/propose Register a new OAuth consumer]. | ||
# As callback URL, use: <code><nowiki>https:/ | # As callback URL, use: <code><nowiki>https://</nowiki><TOOL NAME>.toolforge.org/oauth-callback</code> | ||
# As contact e-mail address, use the e-mail address linked to your Wikimedia unified account. | # As contact e-mail address, use the e-mail address linked to your Wikimedia unified account. | ||
# Keep the default grant settings ('Request authorization for specific permissions.' with just 'Basic rights' selected) | # Keep the default grant settings ('Request authorization for specific permissions.' with just 'Basic rights' selected) | ||
Line 429: | Line 429: | ||
# Copy the ''consumer token'' and ''secret token'' values that are generated. These are needed for the config.yaml file. | # Copy the ''consumer token'' and ''secret token'' values that are generated. These are needed for the config.yaml file. | ||
{{Codesample|lang=shell-session|scheme=light|code= | |||
$ cat >> $HOME/www/python/src/config.yaml << EOF | $ cat >> $HOME/www/python/src/config.yaml << EOF | ||
SECRET_KEY: $(python -c "import os; print repr(os.urandom(24))") | SECRET_KEY: $(python -c "import os; print repr(os.urandom(24))") | ||
Line 436: | Line 436: | ||
CONSUMER_SECRET: the 'secret token' value from your OAuth consumer registration | CONSUMER_SECRET: the 'secret token' value from your OAuth consumer registration | ||
EOF | EOF | ||
}} | |||
;Restart the webservice | |||
{{Codesample|lang=shell-session|scheme=light|code= | |||
$ webservice restart | $ webservice restart | ||
Restarting webservice... | Restarting webservice... | ||
}} | |||
Once the webservice has restarted, navigate to <code><nowiki>https://</nowiki><TOOL NAME>.toolforge.org/</code> in your web browser to see the new landing page. | |||
Try using the ''login'' and ''logout'' links to test out the OAuth integration. | Try using the ''login'' and ''logout'' links to test out the OAuth integration. | ||
Line 449: | Line 450: | ||
=== bash: webservice: command not found === | === bash: webservice: command not found === | ||
# Check shell prompt. | # Check shell prompt. | ||
# If it ends in <code>@interactive $</code>, you are inside a Kubernetes shell (<code>webservice --backend=kubernetes | # If it ends in <code>@interactive $</code>, you are inside a Kubernetes shell (<code>webservice --backend=kubernetes python3.7 shell</code>). | ||
#:* The <code>webservice</code> command is only available on the Toolforge bastions. | #:* The <code>webservice</code> command is only available on the Toolforge bastions. | ||
# Type <code>exit</code> to leave the Kubernetes shell and return to the bastion. | # Type <code>exit</code> to leave the Kubernetes shell and return to the bastion. | ||
=== Error: An error occurred in the OAuth protocol: Invalid signature === | === Error: An error occurred in the OAuth protocol: Invalid signature === | ||
# Double check the values set for <code>CONSUMER_KEY</code> and <code>CONSUMER_SECRET</code> | # Double check the values set for <code>CONSUMER_KEY</code> and <code>CONSUMER_SECRET</code> | ||
=== Get more debugging output from Flask === | === Get more debugging output from Flask === | ||
# Add <code>Debug: True</code> to <code>config.yaml</code> | # Add <code>Debug: True</code> to <code>config.yaml</code> | ||
# Check <code>uwsgi.log</code> for more information. | # Check <code>uwsgi.log</code> for more information. | ||
'''Note:''' This needs a <code>webservice restart</code> to take effect. | '''Note:''' This needs a <code>webservice restart</code> to take effect. | ||
=== toolsws.tool.InvalidToolException: Tool username should begin with tools. === | |||
# Run <code>become <TOOL NAME></code> to change to the tool user. | |||
== Next Steps == | == Next Steps == | ||
Now that your Flask OAuth tool is set-up here are some next steps to consider: | Now that your Flask OAuth tool is set-up here are some next steps to consider: | ||
Line 472: | Line 481: | ||
* Use the [[User:Legoktm/toolforge library|toolforge library]] to set a nice user agent. | * Use the [[User:Legoktm/toolforge library|toolforge library]] to set a nice user agent. | ||
* If possible, run the tool on your local system as well (clone the source code repository, install dependencies, run <code>FLASK_APP=app.py FLASK_ENV=development flask run</code>). In development mode (which you should never run on Toolforge!), Flask offers some additional conveniences, such as automatic reloading of source files on save or an in-browser debugger on errors. | * If possible, run the tool on your local system as well (clone the source code repository, install dependencies, run <code>FLASK_APP=app.py FLASK_ENV=development flask run</code>). In development mode (which you should never run on Toolforge!), Flask offers some additional conveniences, such as automatic reloading of source files on save or an in-browser debugger on errors. | ||
{{:Help:Cloud Services communication}} | |||
== See also == | == See also == | ||
* [[Phab:source/tool-my-first-flask-oauth-tool/|Git repository 'tool-my-first-flask-oauth-tool' on Phabricator]] | * [[Phab:source/tool-my-first-flask-oauth-tool/|Git repository 'tool-my-first-flask-oauth-tool' on Phabricator]] | ||
* [[toolforge:my-first-flask-oauth-tool/|My-first-flask-oauth-tool on Toolforge]] | * [[toolforge:my-first-flask-oauth-tool/|My-first-flask-oauth-tool on Toolforge]] | ||
Line 480: | Line 492: | ||
[[Category:Toolforge]] | [[Category:Toolforge]] | ||
[[Category:Documentation]] | |||
[[Category:Cloud Services]] |
Revision as of 10:38, 10 December 2021
Overview
Python webservices are used by many existing tools. Python is a high-level, interpreted programming language with many available libraries for making webservices and integrating with MediaWiki.
This stub webservice is designed to get a sample Python application installed onto Toolforge as quickly as possible. The application is written using the Flask framework.
The guide will teach you how to:
- Create a new tool
- Run a Python 3 WSGI webservice on Kubernetes
- Allow webservice visitors to authenticate via OAuth v1 using their Wikimedia unified account
Getting started
Prerequisites
Skills
- Basic knowledge of Python
- Basic knowledge of SSH
- Basic knowledge of the Unix command line
- Familiarity with OAuth (v1) terminology would be nice but not required
Accounts
Step-by-step guide
- Step 1: Create a new tool account
- Step 2: Create a basic Flask WSGI webservice
- Step 3: Add a configuration file
- Step 4: Add support for OAuth1 authentication
Step 1: Create a new tool account
- Create a new tool account.
- For the examples in this tutorial,
<TOOL NAME>
is used to indicate places where your unique tool name is used in another command.
- For the examples in this tutorial,
- SSH to login.toolforge.org.
- If your local username is different from your shell account name on Toolforge, you will need to include your Toolforge shell account name in your ssh command (i.e. <shell_username>@login.toolforge.org).
- Run
become <TOOL NAME>
to change to the tool user.
Step 2: Create a basic Flask WSGI webservice
- What is Flask?
Flask is a popular web development framework for Python.
How to create a basic Flask WSGI webservice
- Create the $HOME/www/python/src directory for your application
$ mkdir -p $HOME/www/python/src
- Create a Python virtual environment for the application's external library dependencies
The virtual environment allows the tool to install Python libraries locally without needing a Toolforge administrator's help. The default webservice
configuration will automatically load libraries from $HOME/www/python/venv
.
The webservice will run on Kubernetes. A Kubernetes shell is required to create the virtual environment.
This will ensure that the version of Python the virtual environment uses matches the version of Python used by the Kubernetes runtime.
$ webservice --backend=kubernetes python3.7 shell
If you don't see a command prompt, try pressing enter.
$ python3 -m venv $HOME/www/python/venv
$ source $HOME/www/python/venv/bin/activate
$ pip install --upgrade pip
Downloading/unpacking pip from [...]
[...]
Successfully installed pip
Cleaning up...
If you run python3 -m venv $HOME/www/python/venv
and get an error beginning with "The virtual environment was not created successfully because ensurepip is not available.", you didn't run the first command (the "webservice" one) first.
- Add Flask to the virtual environment
Note: It is Python best practice to use a file named requirements.txt
to keep track of the library dependencies of applications.
$ cat > $HOME/www/python/src/requirements.txt << EOF
flask
EOF
$ pip install -r $HOME/www/python/src/requirements.txt
Collecting flask (from -r www/python/src/requirements.txt (line 1))
[...]
Successfully installed [...]
The initial virtual environment is now set-up. Exit out of the Kubernetes shell and return to the SSH session on the bastion.
$ exit
- Create a 'hello world' WSGI application
# -*- coding: utf-8 -*-
#
# This file is part of the Toolforge flask WSGI tutorial
#
# Copyright (C) 2017 Bryan Davis and contributors
#
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
import flask
app = flask.Flask(__name__)
@app.route('/')
def index():
return 'Hello World!'
Note: The 'Hello World!' file above starts with a license header that places it under the GPLv3+ license.
Code on Toolforge should always be licensed under an Open Source Initiative (OSI) approved license. See the Right to fork policy for more information on this Toolforge policy.
- Start the webservice
$ webservice --backend=kubernetes python3.7 start
Starting webservice.
Once the webservice is started, navigate to https://<TOOL NAME>.toolforge.org/
in your web browser, and see a 'Hello World!' message. It might take a few minutes until it is reachable.
Notes
Toolforge uses a uWSGI configuration designed to make it easy to deploy a typical Python webservice. This configuration uses a 'convention over configuration' design with the following expectations:
- The application will have a WSGI entry point in
$HOME/www/python/src/app.py
in a variable namedapp
. - Python libraries will be loaded from a virtualenv located in
$HOME/www/python/venv
. - Logs will be written to
$HOME/uwsgi.log
- Expected file layout
$HOME
├─ uwsgi.log
└─ www
└─ python
├─ src
│ └─ app.py
└─ venv
Troubleshooting
If there is an error when the webservice is started, look in $HOME/uwsgi.log
and $HOME/error.log
for an explanation.
One Unix utility to use for this is tail
, which will display lines from the end of a file:
$ tail -n 50 $HOME/uwsgi.log
$ tail -n 50 $HOME/error.log
Step 3: Add a configuration file
The application will eventually need some configuration data like OAuth secrets or passwords. These should not be hard coded into the Python files, because the secrets and passwords will be visible once the source code is made public.
There are many different ways to separate code from configuration, but the most straight forward when using Flask is to keep the configuration in a file that can be parsed easily, and then add it to the app.config
object that Flask provides.
How to add a configuration file
- Add PyYAML to the virtual environment
In this tutorial, a YAML file is used to hold secrets. YAML is a nice choice, because it has a simple syntax, is fairly easy for humans to read, and supports both comments and complex values like lists and dictionaries.
Python does not have built in support for parsing YAML files. Install a library to help out.
$ webservice --backend=kubernetes python3.7 shell
If you don't see a command prompt, try pressing enter.
$ source $HOME/www/python/venv/bin/activate
$ cat >> $HOME/www/python/src/requirements.txt << EOF
pyyaml
EOF
$ pip install -r $HOME/www/python/src/requirements.txt
Requirement already satisfied: flask [...]
Collecting pyyaml (from -r req.txt (line 2))
[...]
Successfully installed pyyaml
$ exit
- Read configuration from a file
Update the $HOME/www/python/src/app.py
file to read configuration from a config.yaml
file in the same directory and get the greeting from the configuration file:
# -*- coding: utf-8 -*-
#
# This file is part of the Toolforge Flask + OAuth WSGI tutorial
#
# Copyright (C) 2017 Bryan Davis and contributors
#
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
import flask
import os
import yaml
app = flask.Flask(__name__)
# Load configuration from YAML file
__dir__ = os.path.dirname(__file__)
app.config.update(
yaml.safe_load(open(os.path.join(__dir__, 'config.yaml'))))
@app.route('/')
def index():
return app.config['GREETING']
A configuration file is now required or the application will produce an error. Eventually, secrets will be included in this file. The file's permissions should be changed so that only the tool user can read it.
$ touch $HOME/www/python/src/config.yaml
$ chmod u=rw,go= $HOME/www/python/src/config.yaml
$ cat > $HOME/www/python/src/config.yaml << EOF
GREETING: Goodnight moon!
EOF
Now restart the webservice:
$ webservice restart
Restarting webservice...
Once the webservice has restarted, navigate to https://<TOOL NAME>.toolforge.org/
in your web browser and see the new 'Goodnight moon!' message.
Troubleshooting
If there is an error, look in $HOME/uwsgi.log
and $HOME/error.log
for an explanation.
Step 4: Add support for OAuth authentication
OAuth is a safe mechanism for authenticating a Wikimedia user in the application. For the basics, read more about OAuth on mediawiki.org.
How to add mwoauth to the virtual environment
mwoauth library is used to handle most of the complexity of making OAuth requests to MediaWiki.
$ webservice --backend=kubernetes python3.7 shell
If you don't see a command prompt, try pressing enter.
$ source $HOME/www/python/venv/bin/activate
$ cat >> $HOME/www/python/src/requirements.txt << EOF
mwoauth
EOF
$ pip install -r $HOME/www/python/src/requirements.txt
Requirement already satisfied: flask [...]
Requirement already satisfied: pyyaml [...]
Collecting mwoauth (from -r req.txt (line 3))
[...]
Successfully installed [...]
$ exit
- Update the application code
Here is our new $HOME/www/python/src/app.py
file:
www/python/src/app.py |
---|
The following content has been placed in a collapsed box for improved usability. |
$HOME/www/python/src/app.py # -*- coding: utf-8 -*-
#
# This file is part of the Toolforge Flask + OAuth WSGI tutorial
#
# Copyright (C) 2017 Bryan Davis and contributors
#
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
import flask
import mwoauth
import os
import yaml
app = flask.Flask(__name__)
# Load configuration from YAML file
__dir__ = os.path.dirname(__file__)
app.config.update(
yaml.safe_load(open(os.path.join(__dir__, 'config.yaml'))))
@app.route('/')
def index():
greeting = app.config['GREETING']
username = flask.session.get('username', None)
return flask.render_template(
'index.html', username=username, greeting=greeting)
@app.route('/login')
def login():
"""Initiate an OAuth login.
Call the MediaWiki server to get request secrets and then redirect the
user to the MediaWiki server to sign the request.
"""
consumer_token = mwoauth.ConsumerToken(
app.config['CONSUMER_KEY'], app.config['CONSUMER_SECRET'])
try:
redirect, request_token = mwoauth.initiate(
app.config['OAUTH_MWURI'], consumer_token)
except Exception:
app.logger.exception('mwoauth.initiate failed')
return flask.redirect(flask.url_for('index'))
else:
flask.session['request_token'] = dict(zip(
request_token._fields, request_token))
return flask.redirect(redirect)
@app.route('/oauth-callback')
def oauth_callback():
"""OAuth handshake callback."""
if 'request_token' not in flask.session:
flask.flash(u'OAuth callback failed. Are cookies disabled?')
return flask.redirect(flask.url_for('index'))
consumer_token = mwoauth.ConsumerToken(
app.config['CONSUMER_KEY'], app.config['CONSUMER_SECRET'])
try:
access_token = mwoauth.complete(
app.config['OAUTH_MWURI'],
consumer_token,
mwoauth.RequestToken(**flask.session['request_token']),
flask.request.query_string)
identity = mwoauth.identify(
app.config['OAUTH_MWURI'], consumer_token, access_token)
except Exception:
app.logger.exception('OAuth authentication failed')
else:
flask.session['access_token'] = dict(zip(
access_token._fields, access_token))
flask.session['username'] = identity['username']
return flask.redirect(flask.url_for('index'))
@app.route('/logout')
def logout():
"""Log the user out by clearing their session."""
flask.session.clear()
return flask.redirect(flask.url_for('index'))
|
The above content has been placed in a collapsed box for improved usability. |
The new app.py
uses the Jinja template system that is built into Flask rather than the bare strings that were used in the 'hello world' version. One reason for this is that Jinja will automatically escape strings. This is important in any application that will be serving data gathered from a user or even a database to protect against security vulnerabilities like cross-site scripting.
By default Flask will look for templates in the $HOME/www/python/src/templates
directory.
$ mkdir $HOME/www/python/src/templates
$ edit $HOME/www/python/src/templates/index.html
<!DOCTYPE HTML>
<html>
<head>
<title>My first Flask OAuth tool</title>
</head>
<body>
{% if username %}
<p>Hello {{ username }}!</p>
<p><a href="{{ url_for('logout') }}">logout</a></p>
{% else %}
<p>{{ greeting }}</p>
<p><a href="{{ url_for('login') }}">login</a></p>
{% endif %}
</body>
</html>
- Update the configuration to add OAuth secrets
Add new configuration values to $HOME/www/python/src/config.yaml
file to go with the new code.
- Register a new OAuth consumer.
- As callback URL, use:
https://<TOOL NAME>.toolforge.org/oauth-callback
- As contact e-mail address, use the e-mail address linked to your Wikimedia unified account.
- Keep the default grant settings ('Request authorization for specific permissions.' with just 'Basic rights' selected)
- You will be able to use your own account before the consumer has been approved.
- Copy the consumer token and secret token values that are generated. These are needed for the config.yaml file.
$ cat >> $HOME/www/python/src/config.yaml << EOF
SECRET_KEY: $(python -c "import os; print repr(os.urandom(24))")
OAUTH_MWURI: https://meta.wikimedia.org/w/index.php
CONSUMER_KEY: the 'consumer token' value from your OAuth consumer registration
CONSUMER_SECRET: the 'secret token' value from your OAuth consumer registration
EOF
- Restart the webservice
$ webservice restart
Restarting webservice...
Once the webservice has restarted, navigate to https://<TOOL NAME>.toolforge.org/
in your web browser to see the new landing page.
Try using the login and logout links to test out the OAuth integration.
Additional troubleshooting
bash: webservice: command not found
- Check shell prompt.
- If it ends in
@interactive $
, you are inside a Kubernetes shell (webservice --backend=kubernetes python3.7 shell
).- The
webservice
command is only available on the Toolforge bastions.
- The
- Type
exit
to leave the Kubernetes shell and return to the bastion.
Error: An error occurred in the OAuth protocol: Invalid signature
- Double check the values set for
CONSUMER_KEY
andCONSUMER_SECRET
Get more debugging output from Flask
- Add
Debug: True
toconfig.yaml
- Check
uwsgi.log
for more information.
Note: This needs a webservice restart
to take effect.
toolsws.tool.InvalidToolException: Tool username should begin with tools.
- Run
become <TOOL NAME>
to change to the tool user.
Next Steps
Now that your Flask OAuth tool is set-up here are some next steps to consider:
- Use the OAuth token data stored in
flask.session['access_token']
to make API calls as the authorized user. - Explore the mwclient library to make interacting with the MediaWiki Action API easier.
- Publish your source code in a git repository.
- Add a co-maintainer.
- Create a description page for your tool.
- Use the toolforge library to set a nice user agent.
- If possible, run the tool on your local system as well (clone the source code repository, install dependencies, run
FLASK_APP=app.py FLASK_ENV=development flask run
). In development mode (which you should never run on Toolforge!), Flask offers some additional conveniences, such as automatic reloading of source files on save or an in-browser debugger on errors.
Communication and support
Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia Movement volunteers. Please reach out with questions and join the conversation:
- Chat in real time in the IRC channel #wikimedia-cloud connect, the bridged Telegram group, or the bridged Mattermost channel
- Discuss via email after you subscribed to the cloud@ mailing list