You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Difference between revisions of "Help:Toolforge/My first Flask OAuth tool"

From Wikitech-static
Jump to navigation Jump to search
imported>BryanDavis
(→‎Create a 'hello world' WSGI application: explain OSI acronym and link to OSI site)
imported>Ladsgroup
(This wasted hours of my time debugging why the sessions don't stick)
 
(28 intermediate revisions by 14 users not shown)
Line 1: Line 1:
{{Toolforge nav}}
{{Toolforge nav}}


'''Python webservices''' are used by many existing tools. [[W:Python (programming language)|Python]] is a high-level, interpreted programming language with many available libraries for making webservices and integrating with MediaWiki. This stub webservice is designed to get a sample python application installed onto the tools-project as quickly as possible. The application is written using the [[W:Flask (web framework)|flask framework]]. 
== Overview ==
Python webservices are used by many existing tools. [[W:Python (programming language)|Python]] is a high-level, interpreted programming language with many available libraries for making webservices and integrating with MediaWiki.  


This guide assumes you have [[Help:Toolforge/Getting started|a Toolforge account]] and basic knowledge of Python, [[W:Secure Shell|SSH]] and the UNIX command line.  
This stub webservice is designed to get a sample Python application installed onto Toolforge as quickly as possible. The application is written using the [[W:Flask (web framework)|Flask framework]].


The goal of this guide is to:
'''The guide will teach you how to:'''


* Create a new tool
* Create a new tool
* Run a Python 3 [[W:Web Server Gateway Interface|WSGI]] webservice on [[Help:Toolforge/Web/Kubernetes|Kubernetes]]
* Run a Python 3 [[w:Web Server Gateway Interface|WSGI]] webservice on [[Help:Toolforge/Kubernetes|Kubernetes]]
* Allow webservice visitors to authenticate via [[Mw:Help:OAuth|OAuth]] using their [[W:Wikipedia:Unified login|Wikimedia unified account]]
* Allow webservice visitors to authenticate via [[Mw:Help:OAuth|OAuth]] using their [[W:Wikipedia:Unified login|Wikimedia unified account]]


== Step 1: Create a new tool account ==
== Getting started ==
A '''tool account''' (also known as a 'service group') is a shared UNIX account intended to host and run application code in Toolforge. A tool account can have multiple Toolforge members with 'maintainer' access which allows users to collaborate on building and running the tool.
* Create a [https://toolsadmin.wikimedia.org/tools/ new tool] with a unique name. This name will be part of the URL for the final webservice.[[File:Screenshot about creating a new tool..png|thumb]]
** In this tutorial, we use <code>&lt;TOOL NAME&gt;</code> everywhere the tool name should be used in another command.
* SSH to login.tools.wmflabs.org.
** If you are already logged in, log out and log in again so that your session will see that you have been added to a new tool account.
** If the local and remote username are different. SSH to <shell_username>@login.tools.wmflabs.org.
* Run <code>become &lt;TOOL NAME&gt;</code> to change to the tool user.
** It may take a while for the new tool's home directory and files to get created. If you get an error message like <code>become: no such tool '<TOOL NAME>'</code> wait a few minutes and try again.
** If you get an error message like <code>You are not a member of the group <TOOL NAME></code> try logging out and logging back in again so that your session will see that you have been added to a new tool account.


== Step 2: Create a basic Flask WSGI webservice ==
=== Prerequisites ===


Toolforge has an opinionated default configuration for running WSGI applications. The configuration expects a Python virtual environment in <code>$HOME/www/python/venv</code> and the WSGI application entry point to be named <code>app</code> and loaded from<code>$HOME/www/python/src/app.py</code>. Changing these locations is possible, but outside the scope of this tutorial. Generally it is easier to make your tool conform to the Toolforge expectations than to work around them.
==== Skills ====


'''Expected file layout'''<syntaxhighlight lang="shell">
* Basic knowledge of [[W:Python (programming language)|Python]]
$HOME
* Basic knowledge of [[W:Secure Shell|SSH]]
└── www
* Basic knowledge of the [[W:Unix|Unix command line]]
    └── python
* Familiarity with [[Mw:OAuth/For Developers|OAuth]] terminology would be nice but not required
        ├── src
 
        │  └── app.py
==== Accounts ====
        └── venv
 
</syntaxhighlight>
* [[Help:Getting Started|A Toolforge account]]
 
== Step-by-step guide ==
 
* Step 1: Create a new tool account
* Step 2: Create a basic Flask WSGI webservice
* Step 3: Add a configuration file
* Step 4: Add support for OAuth authentication
 
=== Step 1: Create a new tool account ===
 
# Create [[Portal:Toolforge/Tool Accounts#Create tools with Tool Accounts|a new tool account]].
#* For the examples in this tutorial, <code>&lt;TOOL NAME&gt;</code> is used to indicate places where your unique tool name is used in another command.
# SSH to login.toolforge.org.
#* If your local username is different from your shell account name on Toolforge, you will need to include your Toolforge shell account name in your ssh command (i.e. &lt;shell_username&gt;@login.toolforge.org).
# Run <code>become &lt;TOOL NAME&gt;</code> to change to the tool user.
 
=== Step 2: Create a basic Flask WSGI webservice ===
 
;What is Flask?
 
[[W:Flask (web framework)|Flask]] is a popular web development framework for Python.
 
==== How to create a basic Flask WSGI webservice ====
 
;Create the $HOME/www/python/src directory for your application:


=== Create the $HOME/www/python/src directory for your application ===
{{Codesample|lang=shell-session|scheme=light|code=
<syntaxhighlight lang="shell-session">
$ mkdir -p $HOME/www/python/src
$ mkdir -p $HOME/www/python/src
</syntaxhighlight>
}}
 
;Create a Python virtual environment for the application's external library dependencies
 
The virtual environment allows the tool to install Python libraries locally without needing a Toolforge administrator's help. The default <code>webservice</code> configuration will automatically load libraries from <code>$HOME/www/python/venv</code>.
 
The webservice will run on [[W:Kubernetes|Kubernetes]]. A Kubernetes shell is required to create the virtual environment.


=== Create a Python virtual environment for the application's external library dependencies ===
This will ensure that the version of Python the virtual environment uses matches the version of Python used by the Kubernetes runtime.
The virtual environment will allow your tool to install Python libraries locally without needing a Toolforge administrator's help. The default <code>webservice</code> configuration will automatically load libraries from <code>$HOME/www/python/venv</code>.


We are going to run our webservice on Kubernetes, so we will need to use a Kubernetes shell to create our virtual environment. This will ensure that the version of Python that the virtual environment uses matches the version of Python used by the Kubernetes runtime.<syntaxhighlight lang="shell-session">
{{Codesample|lang=shell-session|scheme=light|code=
$ webservice --backend=kubernetes python shell
$ webservice --backend=kubernetes python3.7 shell
If you don't see a command prompt, try pressing enter.
If you don't see a command prompt, try pressing enter.
$ python3 -m venv $HOME/www/python/venv
$ python3 -m venv $HOME/www/python/venv
Line 53: Line 74:
Successfully installed pip
Successfully installed pip
Cleaning up...
Cleaning up...
</syntaxhighlight>
}}
 
If you run <code>python3 -m venv $HOME/www/python/venv</code> and get an error beginning with "The virtual environment was not created successfully because ensurepip is not available.", you didn't run the first command (the "webservice" one) first.
 
;Add Flask to the virtual environment


=== Add flask to the virtual environment ===
'''Note:''' It is Python best practice to use a file named <code>requirements.txt</code> to keep track of the library dependencies of applications.
Using a file named requirements.txt to keep track of the library dependencies of your application is a Python best practice.<syntaxhighlight lang="shell-session">
 
{{Codesample|lang=shell-session|scheme=light|code=
$ cat > $HOME/www/python/src/requirements.txt << EOF
$ cat > $HOME/www/python/src/requirements.txt << EOF
flask
flask
Line 64: Line 90:
[...]
[...]
Successfully installed [...]
Successfully installed [...]
</syntaxhighlight>We are done setting up the initial virtual environment, so exit out of the Kubernetes shell and return to your SSH session on the bastion.<syntaxhighlight lang="shell-session">
}}
 
The initial virtual environment is now set-up. Exit out of the Kubernetes shell and return to the SSH session on the bastion.
 
{{Codesample|lang=shell-session|scheme=light|code=
$ exit
$ exit
</syntaxhighlight>
}}
 
;Create a 'hello world' WSGI application


=== Create a 'hello world' WSGI application ===
{{Codesample|lang=shell-session|scheme=light|name=$HOME/www/python/src/app.py|code=
Lets make sure that all of the basics are working by creating a very simple 'hello world' WSGI app and running it. The default <code>webservice</code> configuration will look for an <code>app</code> variable in <code>$HOME/www/python/src/app.py</code> as the main WSGI application entry point. Create your <code>$HOME/www/python/src/app.py</code> file with these contents:<syntaxhighlight lang="python">
# -*- coding: utf-8 -*-
# -*- coding: utf-8 -*-
#
#
Line 98: Line 129:
def index():
def index():
   return 'Hello World!'
   return 'Hello World!'
</syntaxhighlight>This file starts with a license header placing it under the [https://www.gnu.org/copyleft/gpl.html GPLv3+ license]. Code on Toolforge should always be licensed under an [https://opensource.org/licenses Open Source Initiative (OSI) approved license]. See the [[Help:Toolforge/Right to fork policy|Right to fork policy]] for more information on the Toolforge policy.
}}
 
'''Note:''' The 'Hello World!' file above starts with a license header that places it under the [https://www.gnu.org/copyleft/gpl.html GPLv3+ license].  
 
Code on Toolforge should always be licensed under an [https://opensource.org/licenses Open Source Initiative (OSI) approved license]. See the [[Help:Toolforge/Right to fork policy|Right to fork policy]] for more information on this Toolforge policy.
 
;Start the webservice


=== Start the webservice ===
{{Codesample|lang=shell-session|scheme=light|code=
<syntaxhighlight lang="shell-session">
$ webservice --backend=kubernetes python3.7 start
$ webservice --backend=kubernetes python start
Starting webservice.
Starting webservice.
</syntaxhighlight>
}}
Once the webservice is started, you should be able to go to <code><nowiki>https://tools.wmflabs.org/</nowiki><TOOL NAME>/</code> in your web browser and see a cheery 'Hello World!' message.
Once the webservice is started, navigate to <code><nowiki>https://</nowiki>&lt;TOOL NAME&gt;.toolforge.org/</code> in your web browser, and see a 'Hello World!' message. It might take a few minutes until it is reachable.


If you see an error instead, look in <code>$HOME/uwsgi.log</code> and <code>$HOME/error.log</code> for an explanation. One unix utility to do that is <code>[https://linux.die.net/man/1/tail tail]</code> which will display lines from the end of a file:
==== Notes ====
<syntaxhighlight lang="shell-session">
Toolforge uses a uWSGI configuration designed to make it easy to deploy a typical Python webservice. This configuration uses a '[[w:Convention over configuration|convention over configuration]]' design with the following expectations:
* The application will have a WSGI entry point in <code>$HOME/www/python/src/app.py</code> in a variable named <code>app</code>.
* Python libraries will be loaded from a virtualenv located in <code>$HOME/www/python/venv</code>.
* Logs will be written to <code>$HOME/uwsgi.log</code>
 
;Expected file layout
{{Codesample|lang=text|scheme=light|code=
$HOME
├─ uwsgi.log
└─ www
    └─ python
        ├─ src
        │  └─ app.py
        └─ venv
}}
 
==== Troubleshooting ====
 
If there is an error when the webservice is started, look in <code>$HOME/uwsgi.log</code> and <code>$HOME/error.log</code> for an explanation.  
 
One Unix utility to use for this is <code>[https://linux.die.net/man/1/tail tail]</code>, which will display lines from the end of a file:
 
{{Codesample|lang=shell-session|scheme=light|code=
$ tail -n 50 $HOME/uwsgi.log
$ tail -n 50 $HOME/uwsgi.log
$ tail -n 50 $HOME/error.log
$ tail -n 50 $HOME/error.log
</syntaxhighlight>
}}
 
=== Step 3: Add a configuration file ===
 
The application will eventually need some configuration data like OAuth secrets or passwords. These should not be hard coded into the Python files, because the secrets and passwords will be visible once the source code is made public.
 
There are many different ways to separate code from configuration, but the most straight forward when using Flask is to keep the configuration in a file that can be parsed easily, and then add it to the <code>app.config</code> object that Flask provides.
 
==== How to add a configuration file ====
 
;Add PyYAML to the virtual environment


== Step 3: Add a configuration file ==
In this tutorial, a YAML file is used to hold secrets. YAML is a nice choice, because it has a simple syntax, is fairly easy for humans to read, and supports both comments and complex values like lists and dictionaries.  
Our application will eventually need some configuration data like OAuth secrets or passwords. These should not be hard coded into the Python files directly because that will make it impossible for us to publish our source code publicly without exposing those secrets.


There are many different ways to separate code from configuration, but the most straight forward when using Flask is to keep our configuration in a file that we can parse easily and then add it to the <code>app.config</code> object that Flask provides.
Python does not have built in support for parsing YAML files. Install a library to help out.


=== Add PyYAML to the virtual environment ===
{{Codesample|lang=shell-session|scheme=light|code=
In this tutorial we will use a YAML file to hold our secrets. YAML is a nice choice because it has a simple syntax, is fairly easy for humans to read, and supports both comments and complex values like lists and dictionaries. Python does not have built in support for parsing YAML files, so we will install a library to help out.<syntaxhighlight lang="shell-session">
$ webservice --backend=kubernetes python3.7 shell
$ webservice --backend=kubernetes python shell
If you don't see a command prompt, try pressing enter.
If you don't see a command prompt, try pressing enter.
$ source $HOME/www/python/venv/bin/activate
$ source $HOME/www/python/venv/bin/activate
Line 132: Line 198:
Successfully installed pyyaml
Successfully installed pyyaml
$ exit
$ exit
</syntaxhighlight>
}}
 
;Read configuration from a file
 
Update the <code>$HOME/www/python/src/app.py</code> file to read configuration from a <code>config.yaml</code> file in the same directory and get the greeting from the configuration file:


=== Read configuration from a file ===
{{Codesample|lang=python3|scheme=light|name=$HOME/www/python/src/app.py|code=
Update our <code>$HOME/www/python/src/app.py</code> file to read configuration from a <code>config.yaml</code> file in the same directory and get the greeting from the configuration file:<syntaxhighlight lang="python">
# -*- coding: utf-8 -*-
# -*- coding: utf-8 -*-
#
#
Line 172: Line 241:
def index():
def index():
     return app.config['GREETING']
     return app.config['GREETING']
</syntaxhighlight>We need a configuration file now or our application will have an error when it tries to read it. We are eventually going to put secrets in this file too, so we need to change the file's permissions so that only our tool user can read it.<syntaxhighlight lang="shell-session">
}}
 
A configuration file is now required or the application will produce an error. Eventually, secrets will be included in this file. The file's permissions should be changed so that only the tool user can read it.
 
{{Codesample|lang=shell-session|scheme=light|code=
$ touch $HOME/www/python/src/config.yaml
$ touch $HOME/www/python/src/config.yaml
$ chmod u=rw,go= $HOME/www/python/src/config.yaml
$ chmod u=rw,go= $HOME/www/python/src/config.yaml
Line 178: Line 251:
GREETING: Goodnight moon!
GREETING: Goodnight moon!
EOF
EOF
</syntaxhighlight>Now restart the webservice:<syntaxhighlight lang="shell-session">
}}
Now restart the webservice:
{{Codesample|lang=shell-session|scheme=light|code=
$ webservice restart
$ webservice restart
Restarting webservice...
Restarting webservice...
</syntaxhighlight>Once the webservice has restarted, you should be able to go to <code><nowiki>https://tools.wmflabs.org/</nowiki><TOOL NAME>/</code> in your web browser and see the new 'Goodnight moon!' message.
}}


If you see an error instead, look in <code>$HOME/uwsgi.log</code> and <code>$HOME/error.log</code> for an explanation.
Once the webservice has restarted, navigate to <code><nowiki>https://</nowiki>&lt;TOOL NAME&gt;.toolforge.org/</code> in your web browser and see the new 'Goodnight moon!' message.


== Step 4: Add support for OAuth authentication ==
==== Troubleshooting ====


OAuth is a safe mechanism for authenticating a Wikimedia user in your application. Explaining how OAuth works and all of the things that a developer should be aware of is out of scope for this tutorial. [[Mw:OAuth/For Developers|Read more about OAuth on mediawiki.org]] if you are unfamiliar with the basics.
If there is an error, look in <code>$HOME/uwsgi.log</code> and <code>$HOME/error.log</code> for an explanation.


=== Add mwoauth to the virtual environment ===
=== Step 4: Add support for OAuth authentication ===
We are going to use the [https://pythonhosted.org/mwoauth/ mwoauth library] to handle most of the complexity of making OAuth requests to MediaWiki.<syntaxhighlight lang="shell-session">
 
$ webservice --backend=kubernetes python shell
OAuth is a safe mechanism for authenticating a Wikimedia user in the application. For the basics, [[Mw:OAuth/For Developers|read more about OAuth on mediawiki.org]].
 
==== How to add mwoauth to the virtual environment ====
 
[https://pythonhosted.org/mwoauth/ mwoauth library] is used to handle most of the complexity of making OAuth requests to MediaWiki.
 
{{Codesample|lang=shell-session|scheme=light|code=
$ webservice --backend=kubernetes python3.7 shell
If you don't see a command prompt, try pressing enter.
If you don't see a command prompt, try pressing enter.
$ source $HOME/www/python/venv/bin/activate
$ source $HOME/www/python/venv/bin/activate
Line 204: Line 286:
Successfully installed [...]
Successfully installed [...]
$ exit
$ exit
</syntaxhighlight>
}}


=== Update the application code ===
;Update the application code
Here is our new <code>$HOME/www/python/src/app.py</code> file:
Here is our new <code>$HOME/www/python/src/app.py</code> file:
{{Collapse top|www/python/src/app.py}}
{{Collapse top|www/python/src/app.py}}
<syntaxhighlight lang="python3">
{{Codesample|lang=python3|scheme=light|name=$HOME/www/python/src/app.py|code=
# -*- coding: utf-8 -*-
# -*- coding: utf-8 -*-
#
#
Line 255: Line 337:
def login():
def login():
     """Initiate an OAuth login.
     """Initiate an OAuth login.
   
     Call the MediaWiki server to get request secrets and then redirect the
     Call the MediaWiki server to get request secrets and then redirect the
     user to the MediaWiki server to sign the request.
     user to the MediaWiki server to sign the request.
Line 291: Line 373:


         identity = mwoauth.identify(
         identity = mwoauth.identify(
             app.config['OAUTH_MWURI'], consumer_token, access_token)
             app.config['OAUTH_MWURI'], consumer_token, access_token)  
     except Exception:
     except Exception:
         app.logger.exception('OAuth authentication failed')
         app.logger.exception('OAuth authentication failed')
   
     else:
     else:
         flask.session['access_token'] = dict(zip(
         flask.session['access_token'] = dict(zip(
Line 308: Line 390:
     flask.session.clear()
     flask.session.clear()
     return flask.redirect(flask.url_for('index'))
     return flask.redirect(flask.url_for('index'))
</syntaxhighlight>
}}
{{Collapse bottom}}
{{Collapse bottom}}


The new <code>app.py</code> uses the Jinja template system that is built into Flask rather than the bare strings that we used in the 'hello world' version. One reason for this is that Jinja will automatically escape strings for us. This is important in any application that will be serving data gathered from a user or even a database to protect against security vulnerabilities like [[w:Cross-site scripting|cross-site scripting]]. By default Flask will look for templates in your <code>$HOME/www/python/src/templates</code> directory.
The new <code>app.py</code> uses the Jinja template system that is built into Flask rather than the bare strings that were used in the 'hello world' version. One reason for this is that Jinja will automatically escape strings. This is important in any application that will be serving data gathered from a user or even a database to protect against security vulnerabilities like [[w:Cross-site scripting|cross-site scripting]].  
<syntaxhighlight lang="shell-session">
 
By default Flask will look for templates in the <code>$HOME/www/python/src/templates</code> directory.
{{Codesample|lang=shell-session|scheme=light|code=
$ mkdir $HOME/www/python/src/templates
$ mkdir $HOME/www/python/src/templates
$ edit $HOME/www/python/src/templates/index.html
$ edit $HOME/www/python/src/templates/index.html
</syntaxhighlight><syntaxhighlight lang="html+jinja">
}}
{{Codesample|lang=html+jinja|scheme=light|name=$HOME/www/python/src/templates/index.html|code=
<!DOCTYPE HTML>
<!DOCTYPE HTML>
<html>
<html>
Line 323: Line 408:
     <body>
     <body>
         {% if username %}
         {% if username %}
         <p>Hello {{ username }}!</p>
         <p>Hello {{((}} username {{))}}!</p>
         <p><a href="{{ url_for('logout') }}">logout</a></p>
         <p><a href="{{((}} url_for('logout') {{))}}">logout</a></p>
         {% else %}
         {% else %}
         <p>{{ greeting }}</p>
         <p>{{((}} greeting {{))}}</p>
         <p><a href="{{ url_for('login') }}">login</a></p>
         <p><a href="{{((}} url_for('login') {{))}}">login</a></p>
         {% endif %}
         {% endif %}
     </body>
     </body>
</html>
</html>
</syntaxhighlight>
}}
 
;Update the configuration to add OAuth secrets


=== Update the configuration to add OAuth secrets ===
Add new configuration values to <code>$HOME/www/python/src/config.yaml</code> file to go with the new code.
We are going to need to add some new configuration values to our <code>$HOME/www/python/src/config.yaml</code> file to go with the new code.  


First we need to [https://meta.wikimedia.org/wiki/Special:OAuthConsumerRegistration/propose register a new OAuth consumer]. That will give us two of the new configuration values we need:
# [https://meta.wikimedia.org/wiki/Special:OAuthConsumerRegistration/propose Register a new OAuth consumer].  
* As callback URL, use: <code><nowiki>https://tools.wmflabs.org/</nowiki>&lt;TOOL NAME&gt;/oauth-callback</code>
# As callback URL, use: <code><nowiki>https://</nowiki>&lt;TOOL NAME&gt;.toolforge.org/oauth-callback</code>
* As contact e-mail address, use the e-mail address linked to your Wikimedia unified account.
# As contact e-mail address, use the e-mail address linked to your Wikimedia unified account.
* Keep the default grant settings ('Request authorization for specific permissions.' with just 'Basic rights' selected)
# Keep the default grant settings ('Request authorization for specific permissions.' with just 'Basic rights' selected)
* Don't worry about approval for now; you can use your own account before the consumer has been approved.
#:* You will be able to use your own account before the consumer has been approved.
* Copy the ''consumer token'' and ''secret token'' values that are generated. You will need them for your config.yaml file.
# Copy the ''consumer token'' and ''secret token'' values that are generated. These are needed for the config.yaml file.


<syntaxhighlight lang="shell-session">
{{Codesample|lang=shell-session|scheme=light|code=
$ cat >> $HOME/www/python/src/config.yaml << EOF
$ cat >> $HOME/www/python/src/config.yaml << EOF
SECRET_KEY: $(python -c "import os; print repr(os.urandom(24))")
SECRET_KEY: $(python -c "import os; print repr(os.urandom(24))")
Line 350: Line 436:
CONSUMER_SECRET: the 'secret token' value from your OAuth consumer registration
CONSUMER_SECRET: the 'secret token' value from your OAuth consumer registration
EOF
EOF
</syntaxhighlight>
}}


Now restart the webservice:<syntaxhighlight lang="shell-session">
;Restart the webservice
{{Codesample|lang=shell-session|scheme=light|code=
$ webservice restart
$ webservice restart
Restarting webservice...
Restarting webservice...
</syntaxhighlight>Once the webservice has restarted, you should be able to go to <code><nowiki>https://tools.wmflabs.org/</nowiki><TOOL NAME>/</code> in your web browser and see the new landing page. Try using the ''login'' and ''logout'' links to test out your OAuth integration.
}}
Once the webservice has restarted, navigate to <code><nowiki>https://</nowiki>&lt;TOOL NAME&gt;.toolforge.org/</code> in your web browser to see the new landing page.


== What next? ==
Try using the ''login'' and ''logout'' links to test out the OAuth integration.
This application is a starting point, but really doesn't do anything interesting yet. The next logical step would be to use the OAuth token data stored in <code>flask.session['access_token']</code> to make [[Mw:API:Main page|API calls]] as the authorized user. You may want to look into [https://pypi.python.org/pypi/mwclient mwclient] library to make interacting with the MediaWiki Action API easier.


Other recommended steps to make your tool compliant with Toolforge policies and easier to maintain:
== Additional troubleshooting ==
* [[Help:Toolforge#Setting up code review and version control|Publish your source code in a git repository]]
* Add a co-maintainer
* [[:Category:Toolforge tools|Create a description page for your tool]]


== Problems? ==
=== bash: webservice: command not found ===
=== bash: webservice: command not found ===
Check to see if your shell prompt ends in <code>@interactive $</code>. If it does, you are inside a Kubernetes shell (<code>webservice --backend=kubernetes python shell</code>). The <code>webservice</code> command is only available on the Toolforge bastions. Type <code>exit</code> to leave the Kubernetes shell and return to the bastion.
 
# Check shell prompt.
# If it ends in <code>@interactive $</code>, you are inside a Kubernetes shell (<code>webservice --backend=kubernetes python3.7 shell</code>).  
#:* The <code>webservice</code> command is only available on the Toolforge bastions.  
# Type <code>exit</code> to leave the Kubernetes shell and return to the bastion.


=== Error: An error occurred in the OAuth protocol: Invalid signature ===
=== Error: An error occurred in the OAuth protocol: Invalid signature ===
Double check the values you set for <code>CONSUMER_KEY</code> and <code>CONSUMER_SECRET</code>
 
# Double check the values set for <code>CONSUMER_KEY</code> and <code>CONSUMER_SECRET</code>


=== Get more debugging output from Flask ===
=== Get more debugging output from Flask ===
Add <code>Debug: True</code> to <code>config.yaml</code> and check <code>uwsgi.log</code> for more information. Note that this needs a <code>webservice restart</code> to take effect.
 
# Add <code>Debug: True</code> to <code>config.yaml</code>  
# Check <code>uwsgi.log</code> for more information.  
'''Note:''' This needs a <code>webservice restart</code> to take effect.
 
=== toolsws.tool.InvalidToolException: Tool username should begin with tools. ===
 
# Run <code>become &lt;TOOL NAME&gt;</code> to change to the tool user.
 
== Next Steps ==
 
Now that your Flask OAuth tool is set-up here are some next steps to consider:
 
* Use the OAuth token data stored in <code>flask.session['access_token']</code> to make [[Mw:API:Main page|API calls]] as the authorized user.
* Explore the [https://pypi.python.org/pypi/mwclient mwclient] library to make interacting with the MediaWiki Action API easier.
* [[Help:Toolforge#Setting up code review and version control|Publish your source code in a git repository]].
* Add a co-maintainer.
* [[:Category:Toolforge tools|Create a description page for your tool]].
* Use the [[User:Legoktm/toolforge library|toolforge library]] to set a nice user agent.
* If possible, run the tool on your local system as well (clone the source code repository, install dependencies, run <code>FLASK_APP=app.py FLASK_ENV=development flask run</code>). In development mode (which you should never run on Toolforge!), Flask offers some additional conveniences, such as automatic reloading of source files on save or an in-browser debugger on errors.
 
{{:Help:Cloud Services communication}}


== See also ==
== See also ==
* [[Phab:source/tool-my-first-flask-oauth-tool/|Git repository 'tool-my-first-flask-oauth-tool' on Phabricator]]
* [[Phab:source/tool-my-first-flask-oauth-tool/|Git repository 'tool-my-first-flask-oauth-tool' on Phabricator]]
* [[toolforge:my-first-flask-oauth-tool/|My-first-flask-oauth-tool on Toolforge]]
* [[toolforge:my-first-flask-oauth-tool/|My-first-flask-oauth-tool on Toolforge]]
* [[Help:Toolforge/My first Django OAuth tool]]
* [[Help:Toolforge/My first Django OAuth tool]]
* [[Help:Toolforge/My first NodeJS OAuth tool]]


[[Category:Toolforge]]
[[Category:Toolforge]]
[[Category:Documentation]]
[[Category:Cloud Services]]

Latest revision as of 17:22, 30 August 2021

Overview

Python webservices are used by many existing tools. Python is a high-level, interpreted programming language with many available libraries for making webservices and integrating with MediaWiki.

This stub webservice is designed to get a sample Python application installed onto Toolforge as quickly as possible. The application is written using the Flask framework.

The guide will teach you how to:

Getting started

Prerequisites

Skills

  • Basic knowledge of Python
  • Basic knowledge of SSH
  • Basic knowledge of the Unix command line
  • Familiarity with OAuth terminology would be nice but not required

Accounts

Step-by-step guide

  • Step 1: Create a new tool account
  • Step 2: Create a basic Flask WSGI webservice
  • Step 3: Add a configuration file
  • Step 4: Add support for OAuth authentication

Step 1: Create a new tool account

  1. Create a new tool account.
    • For the examples in this tutorial, <TOOL NAME> is used to indicate places where your unique tool name is used in another command.
  2. SSH to login.toolforge.org.
    • If your local username is different from your shell account name on Toolforge, you will need to include your Toolforge shell account name in your ssh command (i.e. <shell_username>@login.toolforge.org).
  3. Run become <TOOL NAME> to change to the tool user.

Step 2: Create a basic Flask WSGI webservice

What is Flask?

Flask is a popular web development framework for Python.

How to create a basic Flask WSGI webservice

Create the $HOME/www/python/src directory for your application
$ mkdir -p $HOME/www/python/src
Create a Python virtual environment for the application's external library dependencies

The virtual environment allows the tool to install Python libraries locally without needing a Toolforge administrator's help. The default webservice configuration will automatically load libraries from $HOME/www/python/venv.

The webservice will run on Kubernetes. A Kubernetes shell is required to create the virtual environment.

This will ensure that the version of Python the virtual environment uses matches the version of Python used by the Kubernetes runtime.

$ webservice --backend=kubernetes python3.7 shell
If you don't see a command prompt, try pressing enter.
$ python3 -m venv $HOME/www/python/venv
$ source $HOME/www/python/venv/bin/activate
$ pip install --upgrade pip
Downloading/unpacking pip from [...]
[...]
Successfully installed pip
Cleaning up...

If you run python3 -m venv $HOME/www/python/venv and get an error beginning with "The virtual environment was not created successfully because ensurepip is not available.", you didn't run the first command (the "webservice" one) first.

Add Flask to the virtual environment

Note: It is Python best practice to use a file named requirements.txt to keep track of the library dependencies of applications.

$ cat > $HOME/www/python/src/requirements.txt << EOF
flask
EOF
$ pip install -r $HOME/www/python/src/requirements.txt
Collecting flask (from -r www/python/src/requirements.txt (line 1))
[...]
Successfully installed [...]

The initial virtual environment is now set-up. Exit out of the Kubernetes shell and return to the SSH session on the bastion.

$ exit
Create a 'hello world' WSGI application
$HOME/www/python/src/app.py
# -*- coding: utf-8 -*-
#
# This file is part of the Toolforge flask WSGI tutorial
#
# Copyright (C) 2017 Bryan Davis and contributors
#
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along
# with this program.  If not, see <http://www.gnu.org/licenses/>.

import flask


app = flask.Flask(__name__)


@app.route('/')
def index():
  return 'Hello World!'

Note: The 'Hello World!' file above starts with a license header that places it under the GPLv3+ license.

Code on Toolforge should always be licensed under an Open Source Initiative (OSI) approved license. See the Right to fork policy for more information on this Toolforge policy.

Start the webservice
$ webservice --backend=kubernetes python3.7 start
Starting webservice.

Once the webservice is started, navigate to https://<TOOL NAME>.toolforge.org/ in your web browser, and see a 'Hello World!' message. It might take a few minutes until it is reachable.

Notes

Toolforge uses a uWSGI configuration designed to make it easy to deploy a typical Python webservice. This configuration uses a 'convention over configuration' design with the following expectations:

  • The application will have a WSGI entry point in $HOME/www/python/src/app.py in a variable named app.
  • Python libraries will be loaded from a virtualenv located in $HOME/www/python/venv.
  • Logs will be written to $HOME/uwsgi.log
Expected file layout
$HOME
 ├─ uwsgi.log
 └─ www
     └─ python
         ├─ src
         │   └─ app.py
         └─ venv

Troubleshooting

If there is an error when the webservice is started, look in $HOME/uwsgi.log and $HOME/error.log for an explanation.

One Unix utility to use for this is tail, which will display lines from the end of a file:

$ tail -n 50 $HOME/uwsgi.log
$ tail -n 50 $HOME/error.log

Step 3: Add a configuration file

The application will eventually need some configuration data like OAuth secrets or passwords. These should not be hard coded into the Python files, because the secrets and passwords will be visible once the source code is made public.

There are many different ways to separate code from configuration, but the most straight forward when using Flask is to keep the configuration in a file that can be parsed easily, and then add it to the app.config object that Flask provides.

How to add a configuration file

Add PyYAML to the virtual environment

In this tutorial, a YAML file is used to hold secrets. YAML is a nice choice, because it has a simple syntax, is fairly easy for humans to read, and supports both comments and complex values like lists and dictionaries.

Python does not have built in support for parsing YAML files. Install a library to help out.

$ webservice --backend=kubernetes python3.7 shell
If you don't see a command prompt, try pressing enter.
$ source $HOME/www/python/venv/bin/activate
$ cat >> $HOME/www/python/src/requirements.txt << EOF
pyyaml
EOF
$ pip install -r $HOME/www/python/src/requirements.txt
Requirement already satisfied: flask [...]
Collecting pyyaml (from -r req.txt (line 2))
[...]
Successfully installed pyyaml
$ exit
Read configuration from a file

Update the $HOME/www/python/src/app.py file to read configuration from a config.yaml file in the same directory and get the greeting from the configuration file:

$HOME/www/python/src/app.py
# -*- coding: utf-8 -*-
#
# This file is part of the Toolforge Flask + OAuth WSGI tutorial
#
# Copyright (C) 2017 Bryan Davis and contributors
#
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along
# with this program.  If not, see <http://www.gnu.org/licenses/>.

import flask
import os
import yaml


app = flask.Flask(__name__)


# Load configuration from YAML file
__dir__ = os.path.dirname(__file__)
app.config.update(
    yaml.safe_load(open(os.path.join(__dir__, 'config.yaml'))))


@app.route('/')
def index():
    return app.config['GREETING']

A configuration file is now required or the application will produce an error. Eventually, secrets will be included in this file. The file's permissions should be changed so that only the tool user can read it.

$ touch $HOME/www/python/src/config.yaml
$ chmod u=rw,go= $HOME/www/python/src/config.yaml
$ cat > $HOME/www/python/src/config.yaml << EOF
GREETING: Goodnight moon!
EOF

Now restart the webservice:

$ webservice restart
Restarting webservice...

Once the webservice has restarted, navigate to https://<TOOL NAME>.toolforge.org/ in your web browser and see the new 'Goodnight moon!' message.

Troubleshooting

If there is an error, look in $HOME/uwsgi.log and $HOME/error.log for an explanation.

Step 4: Add support for OAuth authentication

OAuth is a safe mechanism for authenticating a Wikimedia user in the application. For the basics, read more about OAuth on mediawiki.org.

How to add mwoauth to the virtual environment

mwoauth library is used to handle most of the complexity of making OAuth requests to MediaWiki.

$ webservice --backend=kubernetes python3.7 shell
If you don't see a command prompt, try pressing enter.
$ source $HOME/www/python/venv/bin/activate
$ cat >> $HOME/www/python/src/requirements.txt << EOF
mwoauth
EOF
$ pip install -r $HOME/www/python/src/requirements.txt
Requirement already satisfied: flask [...]
Requirement already satisfied: pyyaml [...]
Collecting mwoauth (from -r req.txt (line 3))
[...]
Successfully installed [...]
$ exit
Update the application code

Here is our new $HOME/www/python/src/app.py file:

The new app.py uses the Jinja template system that is built into Flask rather than the bare strings that were used in the 'hello world' version. One reason for this is that Jinja will automatically escape strings. This is important in any application that will be serving data gathered from a user or even a database to protect against security vulnerabilities like cross-site scripting.

By default Flask will look for templates in the $HOME/www/python/src/templates directory.

$ mkdir $HOME/www/python/src/templates
$ edit $HOME/www/python/src/templates/index.html
$HOME/www/python/src/templates/index.html
<!DOCTYPE HTML>
<html>
    <head>
        <title>My first Flask OAuth tool</title>
    </head>
    <body>
        {% if username %}
        <p>Hello {{ username }}!</p>
        <p><a href="{{ url_for('logout') }}">logout</a></p>
        {% else %}
        <p>{{ greeting }}</p>
        <p><a href="{{ url_for('login') }}">login</a></p>
        {% endif %}
    </body>
</html>
Update the configuration to add OAuth secrets

Add new configuration values to $HOME/www/python/src/config.yaml file to go with the new code.

  1. Register a new OAuth consumer.
  2. As callback URL, use: https://<TOOL NAME>.toolforge.org/oauth-callback
  3. As contact e-mail address, use the e-mail address linked to your Wikimedia unified account.
  4. Keep the default grant settings ('Request authorization for specific permissions.' with just 'Basic rights' selected)
    • You will be able to use your own account before the consumer has been approved.
  5. Copy the consumer token and secret token values that are generated. These are needed for the config.yaml file.
$ cat >> $HOME/www/python/src/config.yaml << EOF
SECRET_KEY: $(python -c "import os; print repr(os.urandom(24))")
OAUTH_MWURI: https://meta.wikimedia.org/w/index.php
CONSUMER_KEY: the 'consumer token' value from your OAuth consumer registration
CONSUMER_SECRET: the 'secret token' value from your OAuth consumer registration
EOF
Restart the webservice
$ webservice restart
Restarting webservice...

Once the webservice has restarted, navigate to https://<TOOL NAME>.toolforge.org/ in your web browser to see the new landing page.

Try using the login and logout links to test out the OAuth integration.

Additional troubleshooting

bash: webservice: command not found

  1. Check shell prompt.
  2. If it ends in @interactive $, you are inside a Kubernetes shell (webservice --backend=kubernetes python3.7 shell).
    • The webservice command is only available on the Toolforge bastions.
  3. Type exit to leave the Kubernetes shell and return to the bastion.

Error: An error occurred in the OAuth protocol: Invalid signature

  1. Double check the values set for CONSUMER_KEY and CONSUMER_SECRET

Get more debugging output from Flask

  1. Add Debug: True to config.yaml
  2. Check uwsgi.log for more information.

Note: This needs a webservice restart to take effect.

toolsws.tool.InvalidToolException: Tool username should begin with tools.

  1. Run become <TOOL NAME> to change to the tool user.

Next Steps

Now that your Flask OAuth tool is set-up here are some next steps to consider:

  • Use the OAuth token data stored in flask.session['access_token'] to make API calls as the authorized user.
  • Explore the mwclient library to make interacting with the MediaWiki Action API easier.
  • Publish your source code in a git repository.
  • Add a co-maintainer.
  • Create a description page for your tool.
  • Use the toolforge library to set a nice user agent.
  • If possible, run the tool on your local system as well (clone the source code repository, install dependencies, run FLASK_APP=app.py FLASK_ENV=development flask run). In development mode (which you should never run on Toolforge!), Flask offers some additional conveniences, such as automatic reloading of source files on save or an in-browser debugger on errors.

Communication and support

We communicate and provide support through several primary channels. Please reach out with questions and to join the conversation.

Communicate with us
Connect Best for
Phabricator Workboard #Cloud-Services Task tracking and bug reporting
IRC Channel #wikimedia-cloud connect
Telegram bridge
mattermost bridge
General discussion and support
Mailing List cloud@ Information about ongoing initiatives, general discussion and support
Announcement emails cloud-announce@ Information about critical changes (all messages mirrored to cloud@)
News wiki page News Information about major near-term plans
Cloud Services Blog Clouds & Unicorns Learning more details about some of our work
Wikimedia Technical Blog techblog.wikimedia.org News and stories from the Wikimedia technical movement

See also