You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Help:Toolforge/My first Flask OAuth tool: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>BryanDavis
(Tool Labs -> Toolforge)
imported>Quiddity
(→‎Step 1: Create a new tool account: change NovaServiceGroup to toolsadmin)
Line 13: Line 13:
== Step 1: Create a new tool account ==
== Step 1: Create a new tool account ==
A '''tool account''' (also known as a 'service group') is a shared UNIX account intended to host and run application code in Toolforge. A tool account can have multiple Toolforge members with 'maintainer' access which allows users to collaborate on building and running the tool.
A '''tool account''' (also known as a 'service group') is a shared UNIX account intended to host and run application code in Toolforge. A tool account can have multiple Toolforge members with 'maintainer' access which allows users to collaborate on building and running the tool.
* Create a [https://wikitech.wikimedia.org/w/index.php?title=Special:NovaServiceGroup&action=addservicegroup&projectname=tools new tool] with a unique name. This name will be part of the URL for the final webservice.
* Create a [https://toolsadmin.wikimedia.org/tools/ new tool] with a unique name. This name will be part of the URL for the final webservice.
** In this tutorial, we use <code>&lt;TOOL NAME&gt;</code> everywhere the tool name should be used in another command.
** In this tutorial, we use <code>&lt;TOOL NAME&gt;</code> everywhere the tool name should be used in another command.
* SSH to <code>tools-login.wmflabs.org</code>.
* SSH to <code>tools-login.wmflabs.org</code>.

Revision as of 23:51, 5 September 2017

Python webservices are used by many existing tools. Python is a high-level, interpreted programming language with many available libraries for making webservices and integrating with MediaWiki. This stub webservice is designed to get a sample python application installed onto the tools-project as quickly as possible. The application is written using the flask framework.

This guide assumes you have a Toolforge account and basic knowledge of Python, SSH and the UNIX command line.

The goal of this guide is to:

Step 1: Create a new tool account

A tool account (also known as a 'service group') is a shared UNIX account intended to host and run application code in Toolforge. A tool account can have multiple Toolforge members with 'maintainer' access which allows users to collaborate on building and running the tool.

  • Create a new tool with a unique name. This name will be part of the URL for the final webservice.
    • In this tutorial, we use <TOOL NAME> everywhere the tool name should be used in another command.
  • SSH to tools-login.wmflabs.org.
    • If you are already logged in, log out and log in again so that your session will see that you have been added to a new tool account.
  • Run become <TOOL NAME> to change to the tool user.
    • It may take a while for the new tool's home directory and files to get created. If you get an error message like become: no such tool '<TOOL NAME>' wait a few minutes and try again.
    • If you get an error message like You are not a member of the group <TOOL NAME> try logging out and logging back in again so that your session will see that you have been added to a new tool account.

Step 2: Create a basic Flask WSGI webservice

Toolforge has an opinionated default configuration for running WSGI applications. The configuration expects a Python virtual environment in $HOME/www/python/venv and the WSGI application entry point to be named app and loaded from$HOME/www/python/src/app.py. Changing these locations is possible, but outside the scope of this tutorial. Generally it is easier to make your tool conform to the Toolforge expectations than to work around them.

Expected file layout

$HOME
└── www
    └── python
        ├── src
        │   └── app.py
        └── venv

Create the $HOME/www/python/src directory for your application

$ mkdir -p $HOME/www/python/src

Create a Python virtual environment for the application's external library dependencies

The virtual environment will allow your tool to install Python libraries locally without needing a Toolforge administrator's help. The default webservice configuration will automatically load libraries from $HOME/www/python/venv.

We are going to run our webservice on Kubernetes, so we will need to use a Kubernetes shell to create our virtual environment. This will ensure that the version of Python that the virtual environment uses matches the version of Python used by the Kubernetes runtime.

$ webservice --backend=kubernetes python shell
If you don't see a command prompt, try pressing enter.
$ python3 -m venv $HOME/www/python/venv
$ source $HOME/www/python/venv/bin/activate
$ pip install --upgrade pip
Downloading/unpacking pip from [...]
[...]
Successfully installed pip
Cleaning up...

Add flask to the virtual environment

Using a file named requirements.txt to keep track of the library dependencies of your application is a Python best practice.

$ cat > $HOME/www/python/src/requirements.txt << EOF
flask
EOF
$ pip install -r requirements.txt
Collecting flask (from -r www/python/src/requirements.txt (line 1))
[...]
Successfully installed [...]

We are done setting up the initial virtual environment, so exit out of the Kubernetes shell and return to your SSH session on the bastion.

$ exit

Create a 'hello world' WSGI application

Lets make sure that all of the basics are working by creating a very simple 'hello world' WSGI app and running it. The default webservice configuration will look for an app variable in $HOME/www/python/src/app.py as the main WSGI application entry point. Create your $HOME/www/python/src/app.py file with these contents:

# -*- coding: utf-8 -*-
#
# This file is part of the Toolforge flask WSGI tutorial
#
# Copyright (C) 2017 Bryan Davis and contributors
#
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along
# with this program.  If not, see <http://www.gnu.org/licenses/>.

import flask


app = flask.Flask(__name__)


@app.route('/')
def index():
  return 'Hello World!'

This file starts with a license header placing it under the GPL v3+ license. Code on Toolforge should always be licensed under an OSI approved license. See the Right to fork policy for more information.

Start the webservice

$ webservice --backend=kubernetes python start
Starting webservice.

Once the webservice is started, you should be able to go to https://tools.wmflabs.org/<TOOL NAME>/ in your web browser and see a cheery 'Hello World!' message.

If you see an error instead, look in $HOME/uwsgi.log and $HOME/error.log for an explanation.

Step 3: Add a configuration file

Our application will eventually need some configuration data like OAuth secrets or passwords. These should not be hard coded into the Python files directly because that will make it impossible for us to publish our source code publicly without exposing those secrets.

There are many different ways to separate code from configuration, but the most straight forward when using Flask is to keep our configuration in a file that we can parse easily and then add it to the app.config object that Flask provides.

Add PyYAML to the virtual environment

In this tutorial we will use a YAML file to hold our secrets. YAML is a nice choice because it has a simple syntax, is fairly easy for humans to read, and supports both comments and complex values like lists and dictionaries. Python does not have built in support for parsing YAML files, so we will install a library to help out.

$ webservice --backend=kubernetes python shell
If you don't see a command prompt, try pressing enter.
$ source $HOME/www/python/venv/bin/activate
$ cat >> $HOME/www/python/src/requirements.txt << EOF
pyyaml
EOF
$ pip install -r $HOME/www/python/src/requirements.txt
Requirement already satisfied: flask [...]
Collecting pyyaml (from -r req.txt (line 2))
[...]
Successfully installed pyyaml
$ exit

Read configuration from a file

Update our $HOME/www/python/src/app.py file to read configuration from a config.yaml file in the same directory and get the greeting from the configuration file:

# -*- coding: utf-8 -*-
#
# This file is part of the Toolforge Flask + OAuth WSGI tutorial
#
# Copyright (C) 2017 Bryan Davis and contributors
#
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along
# with this program.  If not, see <http://www.gnu.org/licenses/>.

import flask
import os
import yaml


app = flask.Flask(__name__)


# Load configuration from YAML file
__dir__ = os.path.dirname(__file__)
app.config.update(
    yaml.safe_load(open(os.path.join(__dir__, 'config.yaml'))))


@app.route('/')
def index():
    return app.config['GREETING']

We need a configuration file now or our application will have an error when it tries to read it. We are eventually going to put secrets in this file too, so we need to change the file's permissions so that only our tool user can read it.

$ touch $HOME/www/python/src/config.yaml
$ chmod u=rw,go= $HOME/www/python/src/config.yaml
$ cat > $HOME/www/python/src/config.yaml << EOF
GREETING: Goodnight moon!
EOF

Now restart the webservice:

$ webservice restart
Restarting webservice...

Once the webservice has restarted, you should be able to go to https://tools.wmflabs.org/<TOOL NAME>/ in your web browser and see the new 'Goodnight moon!' message.

If you see an error instead, look in $HOME/uwsgi.log and $HOME/error.log for an explanation.

Step 4: Add support for OAuth authentication

OAuth is a safe mechanism for authenticating a Wikimedia user in your application. Explaining how OAuth works and all of the things that a developer should be aware of is out of scope for this tutorial. Read more about OAuth on mediawiki.org if you are unfamiliar with the basics.

Add mwoauth to the virtual environment

We are going to use the mwoauth library to handle most of the complexity of making OAuth requests to MediaWiki.

$ webservice --backend=kubernetes python shell
If you don't see a command prompt, try pressing enter.
$ source $HOME/www/python/venv/bin/activate
$ cat >> $HOME/www/python/src/requirements.txt << EOF
mwoauth
EOF
$ pip install -r $HOME/www/python/src/requirements.txt
Requirement already satisfied: flask [...]
Requirement already satisfied: pyyaml [...]
Collecting mwoauth (from -r req.txt (line 3))
[...]
Successfully installed [...]
$ exit

Update the application code

Here is our new $HOME/www/python/src/app.py file:

www/python/src/app.py
The following content has been placed in a collapsed box for improved usability. 
# -*- coding: utf-8 -*-
#
# This file is part of the Toolforge Flask + OAuth WSGI tutorial
#
# Copyright (C) 2017 Bryan Davis and contributors
#
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along
# with this program.  If not, see <http://www.gnu.org/licenses/>.

import flask
import mwoauth
import os
import yaml


app = flask.Flask(__name__)


# Load configuration from YAML file
__dir__ = os.path.dirname(__file__)
app.config.update(
    yaml.safe_load(open(os.path.join(__dir__, 'config.yaml'))))


@app.route('/')
def index():
    greeting = app.config['GREETING']
    username = flask.session.get('username', None)
    return flask.render_template(
        'index.html', username=username, greeting=greeting)


@app.route('/login')
def login():
    """Initiate an OAuth login.
	
    Call the MediaWiki server to get request secrets and then redirect the
    user to the MediaWiki server to sign the request.
    """
    consumer_token = mwoauth.ConsumerToken(
        app.config['CONSUMER_KEY'], app.config['CONSUMER_SECRET'])
    try:
        redirect, request_token = mwoauth.initiate(
            app.config['OAUTH_MWURI'], consumer_token)
    except Exception:
        app.logger.exception('mwoauth.initiate failed')
        return flask.redirect(flask.url_for('index'))
    else:
        flask.session['request_token'] = dict(zip(
            request_token._fields, request_token))
        return flask.redirect(redirect)


@app.route('/oauth-callback')
def oauth_callback():
    """OAuth handshake callback."""
    if 'request_token' not in flask.session:
        flask.flash(u'OAuth callback failed. Are cookies disabled?')
        return flask.redirect(flask.url_for('index'))

    consumer_token = mwoauth.ConsumerToken(
        app.config['CONSUMER_KEY'], app.config['CONSUMER_SECRET'])

    try:
        access_token = mwoauth.complete(
            app.config['OAUTH_MWURI'],
            consumer_token,
            mwoauth.RequestToken(**flask.session['request_token']),
            flask.request.query_string)

        identity = mwoauth.identify(
            app.config['OAUTH_MWURI'], consumer_token, access_token)	
    except Exception:
        app.logger.exception('OAuth authentication failed')
	
    else:
        flask.session['access_token'] = dict(zip(
            access_token._fields, access_token))
        flask.session['username'] = identity['username']

    return flask.redirect(flask.url_for('index'))


@app.route('/logout')
def logout():
    """Log the user out by clearing their session."""
    flask.session.clear()
    return flask.redirect(flask.url_for('index'))
The above content has been placed in a collapsed box for improved usability.

The new app.py uses the Jinja template system that is built into Flask rather than the bare strings that we used in the 'hello world' version. One reason for this is that Jinja will automatically escape strings for us. This is important in any application that will be serving data gathered from a user or even a database to protect against security vulnerabilities like cross-site scripting. By default Flask will look for templates in your $HOME/www/python/src/templates directory. 

$ mkdir $HOME/www/python/src/templates
$ edit $HOME/www/python/src/templates/index.html

<!DOCTYPE HTML>
<html>
    <head>
        <title>My first Flask OAuth tool</title>
    </head>
    <body>
        {% if username %}
        <p>Hello {{ username }}!</p>
        <p><a href="{{ url_for('logout') }}">logout</a></p>
        {% else %}
        <p>{{ greeting }}</p>
        <p><a href="{{ url_for('login') }}">login</a></p>
        {% endif %}
    </body>
</html>

Update the configuration to add OAuth secrets

We are going to need to add some new configuration values to our $HOME/www/python/src/config.yaml file to go with the new code.

First we need to register a new OAuth consumer. That will give us two of the new configuration values we need:

  • As callback URL, use: https://tools.wmflabs.org/<TOOL NAME>/oauth-callback
  • As contact e-mail address, use the e-mail address linked to your Wikimedia unified account.
  • Keep the default grant settings ('Request authorization for specific permissions.' with just 'Basic rights' selected)
  • Don't worry about approval for now; you can use your own account before the consumer has been approved.
  • Copy the consumer token and secret token values that are generated. You will need them for your config.yaml file.
$ cat >> $HOME/www/python/src/config.yaml << EOF
SECRET_KEY: $(python -c "import os; print repr(os.urandom(24))")
OAUTH_MWURI: https://meta.wikimedia.org/w/index.php
CONSUMER_KEY: the 'consumer token' value from your OAuth consumer registration
CONSUMER_SECRET: the 'secret token' value from your OAuth consumer registration
EOF

Now restart the webservice:

$ webservice restart
Restarting webservice...

Once the webservice has restarted, you should be able to go to https://tools.wmflabs.org/<TOOL NAME>/ in your web browser and see the new landing page. Try using the login and logout links to test out your OAuth integration.

What next?

This application is a starting point, but really doesn't do anything interesting yet. The next logical step would be to use the OAuth token data stored in flask.session['access_token'] to make API calls as the authorized user. You may want to look into mwclient library to make interacting with the MediaWiki Action API easier.

Other recommended steps to make your tool compliant with Toolforge policies and easier to maintain:

Problems?

bash: webservice: command not found

Check to see if your shell prompt ends in @interactive $. If it does, you are inside a Kubernetes shell (webservice --backend=kubernetes python shell). The webservice command is only available on the Toolforge bastions. Type exit to leave the Kubernetes shell and return to the bastion.

Error: An error occurred in the OAuth protocol: Invalid signature

Double check the values you set for CONSUMER_KEY and CONSUMER_SECRET

Get more debugging output from Flask

Add Debug: True to config.yaml and check uwsgi.log for more information. Note that this needs a webservice restart to take effect.

See also

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Help:Toolforge/My_first_Flask_OAuth_tool&oldid=288181"