You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Difference between revisions of "Help:Toolforge/My first Flask OAuth tool"

From Wikitech-static
Jump to navigation Jump to search
imported>BryanDavis
(Use "Unix" consistently. "UNIX" is a trademark tied to w:Single UNIX Specification compliance.)
imported>BryanDavis
(→‎Notes: update wording and format)
Line 50: Line 50:


#Create a [https://toolsadmin.wikimedia.org/tools/ new tool].   
#Create a [https://toolsadmin.wikimedia.org/tools/ new tool].   
#:* The new tool will need a unique name. The name will become part of the URL for the final webservice, so choose wisely! [[File:Screenshot about creating a new tool..png|thumb]]
#* The new tool will need a unique name. The name will become part of the URL for the final webservice, so choose wisely! [[File:Screenshot about creating a new tool..png|thumb]]
#:* For the examples in this tutorial, <code>&lt;TOOL NAME&gt;</code> is used to indicate places where a unique name would be used in another command.
#* For the examples in this tutorial, <code>&lt;TOOL NAME&gt;</code> is used to indicate places where a unique name would be used in another command.
# SSH to login.tools.wmflabs.org.
# SSH to login.tools.wmflabs.org.
 
#* If your local username is different from your shell account name on Toolforge, you will need to include your Toolforge shell account name in your ssh command (i.e. &lt;shell_username&gt;@login.tools.wmflabs.org). See [[Help:Access]] for more information on using SSH.
=== Notes ===
# Run <code>become &lt;TOOL NAME&gt;</code> to change to the tool user.
 
* <code>&lt;TOOL NAME&gt;</code> must start with a-z, end with a-z or 0-9, be 1-32 characters long, and can only contain lowercase a-z, 0-9, and - characters.
* If you are already logged in, log out and log in again. Your session will see that you have been added to a new tool account.
* If the local and remote username are different, SSH to <shell_username>@login.tools.wmflabs.org.  
* Run <code>become &lt;TOOL NAME&gt;</code> to change to the tool user.


=== Troubleshooting ===
=== Troubleshooting ===
<syntaxhighlight lang="shell-session>
$ become <TOOL NAME>
become: no such tool '<TOOL NAME>'
</syntaxhighlight>
* Make sure you have typed your new tool name correctly.
* It may take a few minutes for your tool's home directory and files to be created. Wait a few minutes, and try again.


'''Possible error messages'''
<syntaxhighlight lang="shell-session>
 
$ become <TOOL NAME>
'''<code>become: no such tool '<TOOL NAME>'</code>'''
You are not a member of the group tools.<TOOL NAME>.
 
Any existing member of the tool's group can add you to that.
* '''Solution:''' Sometimes it takes a while for the new tool's home directory and files to be created. Wait a few minutes, and try again.
</syntaxhighlight>
 
* An active ssh session to login.tools.wmflabs.org will not automatically be updated with new permissions when you are added as a maintainer of a tool. If you are already logged in via ssh when you create the new tool, log out and then log in again to activate your new permissions.
''' <code>You are not a member of the group <TOOL NAME></code>'''
 
* '''Solution:''' Try logging out and logging back in again, so that your session will see you have added a new tool account.


== Step 2: Create a basic Flask WSGI webservice ==
== Step 2: Create a basic Flask WSGI webservice ==
Line 169: Line 167:


=== Notes ===
=== Notes ===
Toolforge uses a uWSGI configuration designed to make it easy to deploy a typical Python webservice. This configuration uses a '[[w:Convention over configuration|convention over configuration]]' design with the following expectations:
* Your application will have a WSGI entry point in <code>$HOME/www/python/src/app.py</code> in a variable named <code>app</code>.
* Python libraries will be loaded from a virtualenv located in <code>$HOME/www/python/venv</code>.
* Logs will be written to <code>$HOME/uwsgi.log</code>


Toolforge uses a strict default configuration for running WSGI applications. The configuration expects a Python virtual environment in <code>$HOME/www/python/venv</code> and the WSGI application entry point to be named <code>app</code> and loaded from<code>$HOME/www/python/src/app.py</code>.
'''Expected file layout'''
 
<pre>
'''Expected file layout'''<syntaxhighlight lang="shell">
$HOME
$HOME
└── www
├─ uwsgi.log
    └── python
└─ www
        ├── src
    └─ python
        │  └── app.py
        ├─ src
        └── venv
        │  └─ app.py
</syntaxhighlight>
        └─ venv
</pre>


=== Troubleshooting ===
=== Troubleshooting ===

Revision as of 19:21, 10 November 2017

Overview

Python webservices are used by many existing tools. Python is a high-level, interpreted programming language with many available libraries for making webservices and integrating with MediaWiki.

This stub webservice is designed to get a sample Python application installed onto Toolforge as quickly as possible. The application is written using the Flask framework.

The guide will teach you how to:

Getting started

Prerequisites

Skills

Accounts

Steps to completion

  • Create a new tool account
  • Create a basic Flask WSGI webservice
  • Add a configuration file
  • Add support for OAuth authentication

Step-by-step guide

Step 1: Create a new tool account

What is a tool account?

A tool account, also known as a "service group," is a shared Unix account. A tool account is used to host and run application code in Toolforge.

A tool account can have multiple Toolforge members with "maintainer" access. This allows users to collaborate together to build and run the tool.

How to create a new tool account

  1. Create a new tool.
    • The new tool will need a unique name. The name will become part of the URL for the final webservice, so choose wisely!
      Screenshot about creating a new tool..png
    • For the examples in this tutorial, <TOOL NAME> is used to indicate places where a unique name would be used in another command.
  2. SSH to login.tools.wmflabs.org.
    • If your local username is different from your shell account name on Toolforge, you will need to include your Toolforge shell account name in your ssh command (i.e. <shell_username>@login.tools.wmflabs.org). See Help:Access for more information on using SSH.
  3. Run become <TOOL NAME> to change to the tool user.

Troubleshooting

$ become <TOOL NAME>
become: no such tool '<TOOL NAME>'
  • Make sure you have typed your new tool name correctly.
  • It may take a few minutes for your tool's home directory and files to be created. Wait a few minutes, and try again.
$ become <TOOL NAME>
You are not a member of the group tools.<TOOL NAME>.
Any existing member of the tool's group can add you to that.
  • An active ssh session to login.tools.wmflabs.org will not automatically be updated with new permissions when you are added as a maintainer of a tool. If you are already logged in via ssh when you create the new tool, log out and then log in again to activate your new permissions.

Step 2: Create a basic Flask WSGI webservice

What is Flask?

Flask is a popular web development framework for Python.

How to create a basic Flask WSGI webservice

Create the $HOME/www/python/src directory for your application

$ mkdir -p $HOME/www/python/src

Create a Python virtual environment for the application's external library dependencies

The virtual environment will allow your tool to install Python libraries locally without needing a Toolforge administrator's help. The default webservice configuration will automatically load libraries from $HOME/www/python/venv.

The webservice will run on Kubernetes, so we will need to use a Kubernetes shell to create our virtual environment.

This will ensure that the version of Python that the virtual environment uses matches the version of Python used by the Kubernetes runtime.

$ webservice --backend=kubernetes python shell
If you don't see a command prompt, try pressing enter.
$ python3 -m venv $HOME/www/python/venv
$ source $HOME/www/python/venv/bin/activate
$ pip install --upgrade pip
Downloading/unpacking pip from [...]
[...]
Successfully installed pip
Cleaning up...

Add Flask to the virtual environment

Note: It is Python best practice to use a file named requirements.txt to keep track of the library dependencies of your application.

$ cat > $HOME/www/python/src/requirements.txt << EOF
flask
EOF
$ pip install -r $HOME/www/python/src/requirements.txt
Collecting flask (from -r www/python/src/requirements.txt (line 1))
[...]
Successfully installed [...]

The initial virtual environment is now set-up. Exit out of the Kubernetes shell and return to your SSH session on the bastion.

$ exit

Create a 'hello world' WSGI application

# -*- coding: utf-8 -*-
#
# This file is part of the Toolforge flask WSGI tutorial
#
# Copyright (C) 2017 Bryan Davis and contributors
#
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along
# with this program.  If not, see <http://www.gnu.org/licenses/>.

import flask


app = flask.Flask(__name__)


@app.route('/')
def index():
  return 'Hello World!'

Note: The 'Hello World!' file above starts with a license header that places it under the GPLv3+ license.

Code on Toolforge should always be licensed under an Open Source Initiative (OSI) approved license. See the Right to fork policy for more information on this Toolforge policy.

Start the webservice

$ webservice --backend=kubernetes python start
Starting webservice.

Once the webservice is started, navigate to https://tools.wmflabs.org/<TOOL NAME>/ in your web browser, and see a 'Hello World!' message.

Notes

Toolforge uses a uWSGI configuration designed to make it easy to deploy a typical Python webservice. This configuration uses a 'convention over configuration' design with the following expectations:

  • Your application will have a WSGI entry point in $HOME/www/python/src/app.py in a variable named app.
  • Python libraries will be loaded from a virtualenv located in $HOME/www/python/venv.
  • Logs will be written to $HOME/uwsgi.log

Expected file layout

$HOME
 ├─ uwsgi.log
 └─ www
     └─ python
         ├─ src
         │   └─ app.py
         └─ venv

Troubleshooting

If you see an error when you start the webservice, look in $HOME/uwsgi.log and $HOME/error.log for an explanation.

One Unix utility to use for this is tail, which will display lines from the end of a file:

$ tail -n 50 $HOME/uwsgi.log
$ tail -n 50 $HOME/error.log

Step 3: Add a configuration file

Your application will eventually need some configuration data like OAuth secrets or passwords. These should not be hard coded into the Python files, because the secrets and passwords will be visible once the source code is made public.

There are many different ways to separate code from configuration, but the most straight forward when using Flask is to keep the configuration in a file that can be parsed easily, and then add it to the app.config object that Flask provides.

How to add a configuration file

Add PyYAML to the virtual environment

In this tutorial, we will use a YAML file to hold our secrets. YAML is a nice choice because it has a simple syntax, is fairly easy for humans to read, and supports both comments and complex values like lists and dictionaries.

Python does not have built in support for parsing YAML files, so we will install a library to help out.

$ webservice --backend=kubernetes python shell
If you don't see a command prompt, try pressing enter.
$ source $HOME/www/python/venv/bin/activate
$ cat >> $HOME/www/python/src/requirements.txt << EOF
pyyaml
EOF
$ pip install -r $HOME/www/python/src/requirements.txt
Requirement already satisfied: flask [...]
Collecting pyyaml (from -r req.txt (line 2))
[...]
Successfully installed pyyaml
$ exit

Read configuration from a file

Update the $HOME/www/python/src/app.py file to read configuration from a config.yaml file in the same directory and get the greeting from the configuration file:

# -*- coding: utf-8 -*-
#
# This file is part of the Toolforge Flask + OAuth WSGI tutorial
#
# Copyright (C) 2017 Bryan Davis and contributors
#
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along
# with this program.  If not, see <http://www.gnu.org/licenses/>.

import flask
import os
import yaml


app = flask.Flask(__name__)


# Load configuration from YAML file
__dir__ = os.path.dirname(__file__)
app.config.update(
    yaml.safe_load(open(os.path.join(__dir__, 'config.yaml'))))


@app.route('/')
def index():
    return app.config['GREETING']

A configuration file is now required or the application will produce an error. Eventually, secrets will be included in this file. The file's permissions should be changed so that only the tool user can read it.

$ touch $HOME/www/python/src/config.yaml
$ chmod u=rw,go= $HOME/www/python/src/config.yaml
$ cat > $HOME/www/python/src/config.yaml << EOF
GREETING: Goodnight moon!
EOF

Now restart the webservice:

$ webservice restart
Restarting webservice...

Once the webservice has restarted, you should be able to go to https://tools.wmflabs.org/<TOOL NAME>/ in your web browser and see the new 'Goodnight moon!' message.

Troubleshooting

If you see an error instead, look in $HOME/uwsgi.log and $HOME/error.log for an explanation.

Step 4: Add support for OAuth authentication

OAuth is a safe mechanism for authenticating a Wikimedia user in your application. If you are unfamiliar with the basics, read more about OAuth on mediawiki.org.

How to add mwoauth to the virtual environment

mwoauth library is used to handle most of the complexity of making OAuth requests to MediaWiki.

$ webservice --backend=kubernetes python shell
If you don't see a command prompt, try pressing enter.
$ source $HOME/www/python/venv/bin/activate
$ cat >> $HOME/www/python/src/requirements.txt << EOF
mwoauth
EOF
$ pip install -r $HOME/www/python/src/requirements.txt
Requirement already satisfied: flask [...]
Requirement already satisfied: pyyaml [...]
Collecting mwoauth (from -r req.txt (line 3))
[...]
Successfully installed [...]
$ exit

Update the application code

Here is our new $HOME/www/python/src/app.py file:

The new app.py uses the Jinja template system that is built into Flask rather than the bare strings that we used in the 'hello world' version. One reason for this is that Jinja will automatically escape strings. This is important in any application that will be serving data gathered from a user or even a database to protect against security vulnerabilities like cross-site scripting.

By default Flask will look for templates in the $HOME/www/python/src/templates directory.

$ mkdir $HOME/www/python/src/templates
$ edit $HOME/www/python/src/templates/index.html
<!DOCTYPE HTML>
<html>
    <head>
        <title>My first Flask OAuth tool</title>
    </head>
    <body>
        {% if username %}
        <p>Hello {{ username }}!</p>
        <p><a href="{{ url_for('logout') }}">logout</a></p>
        {% else %}
        <p>{{ greeting }}</p>
        <p><a href="{{ url_for('login') }}">login</a></p>
        {% endif %}
    </body>
</html>

Update the configuration to add OAuth secrets

Add new configuration values to $HOME/www/python/src/config.yaml file to go with the new code

  1. Register a new OAuth consumer.
  2. As callback URL, use: https://tools.wmflabs.org/<TOOL NAME>/oauth-callback
  3. As contact e-mail address, use the e-mail address linked to your Wikimedia unified account.
  4. Keep the default grant settings ('Request authorization for specific permissions.' with just 'Basic rights' selected)
    • Don't worry about approval for now; you can use your own account before the consumer has been approved.
  5. Copy the consumer token and secret token values that are generated. You will need them for your config.yaml file.
$ cat >> $HOME/www/python/src/config.yaml << EOF
SECRET_KEY: $(python -c "import os; print repr(os.urandom(24))")
OAUTH_MWURI: https://meta.wikimedia.org/w/index.php
CONSUMER_KEY: the 'consumer token' value from your OAuth consumer registration
CONSUMER_SECRET: the 'secret token' value from your OAuth consumer registration
EOF

Restart the webservice

 
$ webservice restart
Restarting webservice...

Once the webservice has restarted, navigate to https://tools.wmflabs.org/<TOOL NAME>/ in your web browser to see the new landing page.

Try using the login and logout links to test out your OAuth integration.

Additional troubleshooting

bash: webservice: command not found

  1. Check shell prompt.
  2. If it ends in @interactive $, you are inside a Kubernetes shell (webservice --backend=kubernetes python shell).
    • The webservice command is only available on the Toolforge bastions.
  3. Type exit to leave the Kubernetes shell and return to the bastion.

Error: An error occurred in the OAuth protocol: Invalid signature

  1. Double check the values you set for CONSUMER_KEY and CONSUMER_SECRET

Get more debugging output from Flask

  1. Add Debug: True to config.yaml
  2. Check uwsgi.log for more information.

Note: This needs a webservice restart to take effect.

Next Steps

Now that your Flask OAuth tool is set-up here are some next steps to consider:

See also