You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Difference between revisions of "Help:Toolforge/My first Django OAuth tool"

From Wikitech-static
Jump to navigation Jump to search
imported>Lucas Werkmeister
(→‎Login and deploy: add instruction to make activate file non-world-readable before adding production secrets to it)
imported>Quiddity
m (fixes)
 
Line 18: Line 18:
First you need to create a Python 3 virtual environment:
First you need to create a Python 3 virtual environment:


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
$ python3 -m venv venv-my-first-django-oauth-app
$ python3 -m venv venv-my-first-django-oauth-app
$ source venv-my-first-django-oauth-app/bin/activate
$ source venv-my-first-django-oauth-app/bin/activate
$ pip install django
$ pip install django
</source>
</syntaxhighlight>


Then set up a new Django project:
Then set up a new Django project:


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
$ mkdir my-first-django-oauth-app
$ mkdir my-first-django-oauth-app
$ cd my-first-django-oauth-app
$ cd my-first-django-oauth-app
$ mkdir src
$ mkdir src
$ django-admin startproject oauth_app src
$ django-admin startproject oauth_app src
</source>
</syntaxhighlight>


Make sure you change the permissions on your settings.py file so that you don't share your app's secrets unintentionally!
Make sure you change the permissions on your settings.py file so that you don't share your app's secrets unintentionally!
<source lang="shell-session">
<syntaxhighlight lang="shell-session">
$ chmod o-r src/oauth_app/settings.py
$ chmod o-r src/oauth_app/settings.py
</source>
</syntaxhighlight>


If you are checking this into source control, you should avoid checking in the settings.py file. If using git, you can use a .gitignore file at the root of your git repo with something like these contents:
If you are checking this into source control, you should avoid checking in the settings.py file. If using git, you can use a .gitignore file at the root of your git repo with something like these contents:
<source lang="bash>
<syntaxhighlight lang="bash>
*settings.py
*settings.py
*.pyc
*.pyc
__pycache__/
__pycache__/
</source>
</syntaxhighlight>


Now we have the main app set up and the project folder structure will look like this:
Now we have the main app set up and the project folder structure will look like this:


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
[my-first-django-oauth-app]$ tree
[my-first-django-oauth-app]$ tree
.
.
Line 57: Line 57:
         ├── urls.py
         ├── urls.py
         └── wsgi.py
         └── wsgi.py
</source>
</syntaxhighlight>


Next we need to add an app for for our actual webpage:
Next we need to add an app for for our actual webpage:


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
$ cd src
$ cd src
$ django-admin startapp user_profile
$ django-admin startapp user_profile
</source>
</syntaxhighlight>


The structure will then look like this:
The structure will then look like this:


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
[my-first-django-oauth-app]$ tree
[my-first-django-oauth-app]$ tree
.
.
Line 87: Line 87:
         ├── tests.py
         ├── tests.py
         └── views.py
         └── views.py
</source>
</syntaxhighlight>


Then add this new app to the installed apps in settings.py:
Then add this new app to the installed apps in settings.py:


<source lang="python">
<syntaxhighlight lang="python">
INSTALLED_APPS = [
INSTALLED_APPS = [
     ...
     ...
     'user_profile',
     'user_profile',
]
]
</source>
</syntaxhighlight>


And then route the main page from the main app to our new <code>user_profile</code> app. This is done in the <code>urls.py</code> file in the main app:
And then route the main page from the main app to our new <code>user_profile</code> app. This is done in the <code>urls.py</code> file in the main app:


<source lang="python">
<syntaxhighlight lang="python">
from django.urls import path, include
from django.urls import path, include
from django.contrib import admin
from django.contrib import admin
Line 108: Line 108:
     path('', include('user_profile.urls')),
     path('', include('user_profile.urls')),
]
]
</source>
</syntaxhighlight>


Create a <code>urls.py</code> in your <code>user_profile</code> folder as well, and add the following to it:
Create a <code>urls.py</code> in your <code>user_profile</code> folder as well, and add the following to it:


<source lang="python">
<syntaxhighlight lang="python">
from django.urls import path
from django.urls import path
from user_profile import views
from user_profile import views
Line 119: Line 119:
     path('', views.index),
     path('', views.index),
]
]
</source>
</syntaxhighlight>


The <code>index</code> view is still missing. Create it in <code>views.py</code>:
The <code>index</code> view is still missing. Create it in <code>views.py</code>:


<source lang="python">
<syntaxhighlight lang="python">
from django.shortcuts import render
from django.shortcuts import render


Line 129: Line 129:
     context = {}
     context = {}
     return render(request, 'user_profile/index.dtl', context)
     return render(request, 'user_profile/index.dtl', context)
</source>
</syntaxhighlight>


Now we only need to create the Django template in the folder <code>templates/user_profile/index.dtl</code> (You can also use html as file extension if you don't get syntax highlighting for dtl files).
Now we only need to create the Django template in the folder <code>templates/user_profile/index.dtl</code> (You can also use html as file extension if you don't get syntax highlighting for dtl files).


<source lang="html">
<syntaxhighlight lang="html">
<!DOCTYPE html>
<!DOCTYPE html>
<html>
<html>
Line 140: Line 140:
</body>
</body>
</html>
</html>
</source>
</syntaxhighlight>


The current file structure will look something like this:
The current file structure will look something like this:


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
[my-first-django-oauth-app]$ tree
[my-first-django-oauth-app]$ tree
.
.
Line 168: Line 168:
         ├── urls.py
         ├── urls.py
         └── views.py
         └── views.py
</source>
</syntaxhighlight>


Now you can start Django's built in development webserver:
Now you can start Django's built in development webserver:


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
$ python manage.py runserver 127.0.0.1:8080
$ python manage.py runserver 127.0.0.1:8080
</source>
</syntaxhighlight>


This will show you the template you just created. Note that we are running on port 8080 because port 8000 is used on some systems. In order to stay consistent we will stick with port 8080 for local development. Because the port is also part of the OAuth callback URL, we will reduce the number of consumers we need to register for development.
This will show you the template you just created. Note that we are running on port 8080 because port 8000 is used on some systems. In order to stay consistent we will stick with port 8080 for local development. Because the port is also part of the OAuth callback URL, we will reduce the number of consumers we need to register for development.
Line 184: Line 184:
Install the package for your virtual environment:
Install the package for your virtual environment:


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
$ pip install social-auth-app-django
$ pip install social-auth-app-django
</source>
</syntaxhighlight>


Add the following in your main app's settings.py:
Add the following in your main app's settings.py:


<source lang="python">
<syntaxhighlight lang="python">
INSTALLED_APPS = [
INSTALLED_APPS = [
     ...
     ...
Line 218: Line 218:
     'django.contrib.auth.backends.ModelBackend',
     'django.contrib.auth.backends.ModelBackend',
)
)
</source>
</syntaxhighlight>


Then we need to add settings for the OAuth provider. You can register your application, after reading the [[Mw:OAuth/For Developers|OAuth for developers]] documentation.
Then we need to add settings for the OAuth provider. You can register your application, after reading the [[Mw:OAuth/For Developers|OAuth for developers]] documentation.
Line 242: Line 242:
The ''consumer token'' is your SOCIAL_AUTH_MEDIAWIKI_KEY, and the ''secret token'' is your SOCIAL_AUTH_MEDIAWIKI_SECRET, in settings.py:
The ''consumer token'' is your SOCIAL_AUTH_MEDIAWIKI_KEY, and the ''secret token'' is your SOCIAL_AUTH_MEDIAWIKI_SECRET, in settings.py:


<source lang="python">
<syntaxhighlight lang="python">
SOCIAL_AUTH_MEDIAWIKI_KEY = 'xxxxxxxxxxxxxxxxxxxxxxxx'
SOCIAL_AUTH_MEDIAWIKI_KEY = 'xxxxxxxxxxxxxxxxxxxxxxxx'
SOCIAL_AUTH_MEDIAWIKI_SECRET = 'xxxxxxxxxxxxxxxxxxxxxxxx'
SOCIAL_AUTH_MEDIAWIKI_SECRET = 'xxxxxxxxxxxxxxxxxxxxxxxx'
SOCIAL_AUTH_MEDIAWIKI_URL = 'https://meta.wikimedia.org/w/index.php'
SOCIAL_AUTH_MEDIAWIKI_URL = 'https://meta.wikimedia.org/w/index.php'
SOCIAL_AUTH_MEDIAWIKI_CALLBACK = 'http://127.0.0.1:8080/oauth/complete/mediawiki/'
SOCIAL_AUTH_MEDIAWIKI_CALLBACK = 'http://127.0.0.1:8080/oauth/complete/mediawiki/'
</source>
</syntaxhighlight>


''N.B. The trailing '/' at the end of the callback is important. Login will silently fail without it.''
''N.B. The trailing '/' at the end of the callback is important. Login will silently fail without it.''
Line 253: Line 253:
After that, you need to apply the migrations in order to create the user models in the database.
After that, you need to apply the migrations in order to create the user models in the database.


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
$ python manage.py migrate
$ python manage.py migrate
</source>
</syntaxhighlight>


Next we need to up a few things for the profile view and the login. Update your <code>user_profile</code> apps <code>urls.py</code> to say something like this:
Next we need to up a few things for the profile view and the login. Update your <code>user_profile</code> apps <code>urls.py</code> to say something like this:


<source lang="python">
<syntaxhighlight lang="python">
from django.urls import path, include
from django.urls import path, include
from user_profile import views
from user_profile import views
Line 269: Line 269:
     path('', views.index),
     path('', views.index),
]
]
</source>
</syntaxhighlight>


In <code>settings.py</code> we need to add the URL pattern named "login" as the <code>LOGIN_URL</code> and the URL pattern "profile" for <code>LOGIN_REDIRECT_URL</code>:
In <code>settings.py</code> we need to add the URL pattern named "login" as the <code>LOGIN_URL</code> and the URL pattern "profile" for <code>LOGIN_REDIRECT_URL</code>:


<source lang="python">
<syntaxhighlight lang="python">
LOGIN_URL = 'login'
LOGIN_URL = 'login'
LOGIN_REDIRECT_URL = 'profile'
LOGIN_REDIRECT_URL = 'profile'
</source>
</syntaxhighlight>


Create two new views for the profile and the login:
Create two new views for the profile and the login:


<source lang="python">
<syntaxhighlight lang="python">
from django.shortcuts import render
from django.shortcuts import render
from django.contrib.auth.decorators import login_required
from django.contrib.auth.decorators import login_required
Line 297: Line 297:
     context = {}
     context = {}
     return render(request, 'user_profile/login.dtl', context)
     return render(request, 'user_profile/login.dtl', context)
</source>
</syntaxhighlight>


Then create the <code>login.dtl</code> file:
Then create the <code>login.dtl</code> file:


<source lang="html">
<syntaxhighlight lang="html">
<!DOCTYPE html>
<!DOCTYPE html>
<html>
<html>
Line 309: Line 309:
</body>
</body>
</html>
</html>
</source>
</syntaxhighlight>


And the <code>profile.dtl</code> file:
And the <code>profile.dtl</code> file:


<source lang="html">
<syntaxhighlight lang="html">
<!DOCTYPE html>
<!DOCTYPE html>
<html>
<html>
Line 321: Line 321:
</body>
</body>
</html>
</html>
</source>
</syntaxhighlight>


And finally update your <code>index.dtl</code> to say:
And finally update your <code>index.dtl</code> to say:


<source lang="html">
<syntaxhighlight lang="html">
<!DOCTYPE html>
<!DOCTYPE html>
<html>
<html>
Line 333: Line 333:
</body>
</body>
</html>
</html>
</source>
</syntaxhighlight>


When you now restart the server and click on the "Profile" link you will be asked to give permission to your newly created OAuth consumer. After approving the consumer you will be redirected to the user page which will print the Wikimedia user name.
When you now restart the server and click on the "Profile" link you will be asked to give permission to your newly created OAuth consumer. After approving the consumer you will be redirected to the user page which will print the Wikimedia user name.
Line 347: Line 347:
In order to make your project work in both environments and keep your secret keys out of source control you can for example define the following variables in <code>venv-my-first-django-oauth-app/bin/activate</code>. That means the keys will be available only if the environment is active:
In order to make your project work in both environments and keep your secret keys out of source control you can for example define the following variables in <code>venv-my-first-django-oauth-app/bin/activate</code>. That means the keys will be available only if the environment is active:


<source lang="bash">
<syntaxhighlight lang="bash">
...
...
export django_secret="very-secret-key"
export django_secret="very-secret-key"
Line 353: Line 353:
export mediawiki_secret="very-secret-mediawiki-secret"
export mediawiki_secret="very-secret-mediawiki-secret"
export mediawiki_callback="http://127.0.0.1:8080/oauth/complete/mediawiki/"
export mediawiki_callback="http://127.0.0.1:8080/oauth/complete/mediawiki/"
</source>
</syntaxhighlight>


And after reactivating the venv you will be able to access the variables in <code>settings.py</code>
And after reactivating the venv you will be able to access the variables in <code>settings.py</code>


<source lang="python">
<syntaxhighlight lang="python">
SECRET_KEY = os.environ.get('django_secret')
SECRET_KEY = os.environ.get('django_secret')
...
...
Line 364: Line 364:
SOCIAL_AUTH_MEDIAWIKI_URL = 'https://meta.wikimedia.org/w/index.php'
SOCIAL_AUTH_MEDIAWIKI_URL = 'https://meta.wikimedia.org/w/index.php'
SOCIAL_AUTH_MEDIAWIKI_CALLBACK = os.environ.get('mediawiki_callback')
SOCIAL_AUTH_MEDIAWIKI_CALLBACK = os.environ.get('mediawiki_callback')
</source>
</syntaxhighlight>


Now you can safely add our project to git. Run this in your project directory:
Now you can safely add our project to git. Run this in your project directory:


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
$ git init
$ git init
</source>
</syntaxhighlight>


You can then also add your requirements to your project:
You can then also add your requirements to your project:


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
$ pip freeze > requirements.txt
$ pip freeze > requirements.txt
</source>
</syntaxhighlight>


We also need to update the allowed hosts (See: [https://docs.djangoproject.com/en/dev/ref/settings/#allowed-hosts Allowed-hosts]) in <code>settings.py</code>:
We also need to update the allowed hosts (See: [https://docs.djangoproject.com/en/dev/ref/settings/#allowed-hosts Allowed-hosts]) in <code>settings.py</code>:


<source lang="python">
<syntaxhighlight lang="python">
ALLOWED_HOSTS = [
ALLOWED_HOSTS = [
     'toolforge.org',
     'toolforge.org',
]
]
</source>
</syntaxhighlight>


Don't forget to create a <code>.gitignore</code> file that excludes all files that should not be under source control. For example:
Don't forget to create a <code>.gitignore</code> file that excludes all files that should not be under source control. For example:
Line 399: Line 399:
On Toolforge your Django app will run with UWSGI. In order to have a similar setup on the development machine we will create a similar setup using Nginx. First we need to create an <code>app.ini</code> in our <code>src</code> directory:
On Toolforge your Django app will run with UWSGI. In order to have a similar setup on the development machine we will create a similar setup using Nginx. First we need to create an <code>app.ini</code> in our <code>src</code> directory:


<source lang="ini">
<syntaxhighlight lang="ini">
[uwsgi]
[uwsgi]
module = oauth_app.wsgi
module = oauth_app.wsgi
Line 418: Line 418:


die-on-term = true
die-on-term = true
</source>
</syntaxhighlight>


You should be able to start the uwsgi file on its own by using: <code>uwsgi app.ini</code>. If it fails this is often a permission error with the <code>/run/uwsgi</code> directory. The current user needs to be able to create the socket file in this directory.
You should be able to start the uwsgi file on its own by using: <code>uwsgi app.ini</code>. If it fails this is often a permission error with the <code>/run/uwsgi</code> directory. The current user needs to be able to create the socket file in this directory.
Line 424: Line 424:
For this file we will create a service in <code>/etc/systemd/system/django-oauth.uwsgi.service</code>:
For this file we will create a service in <code>/etc/systemd/system/django-oauth.uwsgi.service</code>:


<source lang="ini">
<syntaxhighlight lang="ini">
[Unit]
[Unit]
Description=uWSGI instance to serve django oauth app
Description=uWSGI instance to serve django oauth app
Line 434: Line 434:
[Install]
[Install]
WantedBy=multi-user.target
WantedBy=multi-user.target
</source>
</syntaxhighlight>


Ubuntu for example does not have <code>/usr/bin/bash</code> and you need to use <code>/bin/bash</code>. Errors like this can be seen when using <code>sudo systemctl status django-oauth.uwsgi.service</code>.
Ubuntu for example does not have <code>/usr/bin/bash</code> and you need to use <code>/bin/bash</code>. Errors like this can be seen when using <code>sudo systemctl status django-oauth.uwsgi.service</code>.
Line 440: Line 440:
You can then start this service by running <code>sudo systemctl start django-oauth.uwsgi.service</code>. If that has worked you can continue to configure the webserver. Go to <code>/etc/nginx/nginx.conf</code> and add the following configuration:
You can then start this service by running <code>sudo systemctl start django-oauth.uwsgi.service</code>. If that has worked you can continue to configure the webserver. Go to <code>/etc/nginx/nginx.conf</code> and add the following configuration:


<source lang="nginx">
<syntaxhighlight lang="nginx">
http {
http {


Line 458: Line 458:


     server { ...
     server { ...
</source>
</syntaxhighlight>


After that you can either restart or start the nginx server (<code>sudo systemctl start nginx</code>). You can check if everything is working correctly by entering: <code>sudo systemctl status nginx</code>. On an SELinux enabled system you might still need to set the correct target context for the <code>sock</code> file. Now go to <code>127.0.0.1:8080</code> and see if your page still loads correctly.
After that you can either restart or start the nginx server (<code>sudo systemctl start nginx</code>). You can check if everything is working correctly by entering: <code>sudo systemctl status nginx</code>. On an SELinux enabled system you might still need to set the correct target context for the <code>sock</code> file. Now go to <code>127.0.0.1:8080</code> and see if your page still loads correctly.
Line 466: Line 466:
After configuring two-factor authentication you can log into Toolforge using:
After configuring two-factor authentication you can log into Toolforge using:


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
$ ssh YOUR_USER_NAME@login.toolforge.org
$ ssh YOUR_USER_NAME@login.toolforge.org
</source>
</syntaxhighlight>


After the successful login you need to switch to your new tool:
After the successful login you need to switch to your new tool:


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
$ become YOUR_TOOL_NAME
$ become YOUR_TOOL_NAME
</source>
</syntaxhighlight>


You can use the following commands to control the web server:
You can use the following commands to control the web server:
<source lang="shell-session">
<syntaxhighlight lang="shell-session">
$ webservice --backend=kubernetes python status/start/stop/restart
$ webservice --backend=kubernetes python status/start/stop/restart
</source>
</syntaxhighlight>


Create the following file structure for the web server config file:
Create the following file structure for the web server config file:
Line 493: Line 493:
In the <code>uwsgi.ini</code> file add the following:
In the <code>uwsgi.ini</code> file add the following:


<source lang="ini">
<syntaxhighlight lang="ini">
[uwsgi]
[uwsgi]
check-static = /data/project/YOUR_TOOL_NAME/www/python/static
check-static = /data/project/YOUR_TOOL_NAME/www/python/static
</source>
</syntaxhighlight>


Any files in this static directory will be publicly accessible on the internet.
Any files in this static directory will be publicly accessible on the internet.
Line 502: Line 502:
In the <code>python</code> directory you can clone your git repository (The dot at the end clones into your current directory instead of creating another subfolder):
In the <code>python</code> directory you can clone your git repository (The dot at the end clones into your current directory instead of creating another subfolder):


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
[python] $ git clone ADDRESS_OF_REPOSITORY .
[python] $ git clone ADDRESS_OF_REPOSITORY .
</source>
</syntaxhighlight>


You also have to create a virtual environment on the server using:
You also have to create a virtual environment on the server using:


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
[python] $ virtualenv -p /usr/bin/python3.4 venv
[python] $ virtualenv -p /usr/bin/python3.4 venv
</source>
</syntaxhighlight>


Then you are ready to request your production keys (which you need to be even more careful about not committing):
Then you are ready to request your production keys (which you need to be even more careful about not committing):
Line 524: Line 524:
Make the virtual environment activate file non-world-readable to ensure that nobody else will be able to access the production keys in it:
Make the virtual environment activate file non-world-readable to ensure that nobody else will be able to access the production keys in it:


<source lang=shell>
<syntaxhighlight lang=shell>
[python] $ chmod go-rwx venv/bin/activate
[python] $ chmod go-rwx venv/bin/activate
</source>
</syntaxhighlight>


Then edit the file to add your production keys to it:
Then edit the file to add your production keys to it:


<source lang="shell">
<syntaxhighlight lang="shell">
export django_secret="very-secret-production-key"
export django_secret="very-secret-production-key"
export mediawiki_key="very-secret-mediawiki-production-key"
export mediawiki_key="very-secret-mediawiki-production-key"
export mediawiki_secret="very-secret-mediawiki-production-secret"
export mediawiki_secret="very-secret-mediawiki-production-secret"
export mediawiki_callback="https://YOUR-TOOL-NAME.toolforge.org/oauth/complete/mediawiki/"
export mediawiki_callback="https://YOUR-TOOL-NAME.toolforge.org/oauth/complete/mediawiki/"
</source>
</syntaxhighlight>


Activate the environment to install the dependencies and apply the migrations, then deactivate it again:
Activate the environment to install the dependencies and apply the migrations, then deactivate it again:


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
[python] $ source venv/bin/activate
[python] $ source venv/bin/activate
[python] $ pip install -r requirements.txt
[python] $ pip install -r requirements.txt
Line 547: Line 547:


[python] $ deactivate
[python] $ deactivate
</source>
</syntaxhighlight>


Running the 'migrate' command will also create the database as a sqlite file. The file with our current settings is at <code>/www/python/src/db.sqlite3</code>. You should take care to back up this file and be confidential with the information that is stored within it.
Running the 'migrate' command will also create the database as a sqlite file. The file with our current settings is at <code>/www/python/src/db.sqlite3</code>. You should take care to back up this file and be confidential with the information that is stored within it.
Line 553: Line 553:
With the current setup you also have to create a 2nd <code>app.py</code> in <code>~/www/python/src</code> where the standard webservice will look for it. It also expects the variable to be named <code>app</code>:
With the current setup you also have to create a 2nd <code>app.py</code> in <code>~/www/python/src</code> where the standard webservice will look for it. It also expects the variable to be named <code>app</code>:


<source lang="python">
<syntaxhighlight lang="python">
import os
import os


Line 561: Line 561:


app = get_wsgi_application()
app = get_wsgi_application()
</source>
</syntaxhighlight>


After making these configurations you should be able to start the webserver using:
After making these configurations you should be able to start the webserver using:


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
$ webservice --backend=kubernetes python start
$ webservice --backend=kubernetes python start
</source>
</syntaxhighlight>


In case there are any problems with the configuration they will appear in the tools root directory in the file: <code>~/uwsgi.log</code>.
In case there are any problems with the configuration they will appear in the tools root directory in the file: <code>~/uwsgi.log</code>.
Line 573: Line 573:
The project structure as of now will look like this:
The project structure as of now will look like this:


<source lang="shell-session">
<syntaxhighlight lang="shell-session">
$ tree
$ tree
.
.
Line 609: Line 609:
         ├── uwsgi.ini
         ├── uwsgi.ini
         └── venv
         └── venv
</source>
</syntaxhighlight>


You can view a working example of the OAuth app at:
You can view a working example of the OAuth app at:

Latest revision as of 19:17, 4 September 2021

Warning Caution: This page may contain inaccuracies. It is currently being edited and redesigned for better readability. For further information, please see: Phab:T245683 and Phab:T198508.

Overview

This guide will show how to set up a Django app on Toolforge. We will use the python-social-auth library to implement OAuth authentification with Wikipedia, using the Mediawiki OAuth capabilities.

If you are new to Django you should read through some of the tutorials in the Tutorials section first.

Local development and testing

First, we will set the project up on your local machine for development.

Create your Django project

Most of the setup is not different from the many tutorials that you can find for Django.

First you need to create a Python 3 virtual environment:

$ python3 -m venv venv-my-first-django-oauth-app
$ source venv-my-first-django-oauth-app/bin/activate
$ pip install django

Then set up a new Django project:

$ mkdir my-first-django-oauth-app
$ cd my-first-django-oauth-app
$ mkdir src
$ django-admin startproject oauth_app src

Make sure you change the permissions on your settings.py file so that you don't share your app's secrets unintentionally!

$ chmod o-r src/oauth_app/settings.py

If you are checking this into source control, you should avoid checking in the settings.py file. If using git, you can use a .gitignore file at the root of your git repo with something like these contents:

*settings.py
*.pyc
__pycache__/

Now we have the main app set up and the project folder structure will look like this:

[my-first-django-oauth-app]$ tree
.
└── src
    ├── manage.py
    └── oauth_app
        ├── __init__.py
        ├── settings.py
        ├── urls.py
        └── wsgi.py

Next we need to add an app for for our actual webpage:

$ cd src
$ django-admin startapp user_profile

The structure will then look like this:

[my-first-django-oauth-app]$ tree
.
└── src
    ├── manage.py
    ├── oauth_app
    │   ├── __init__.py
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    └── user_profile
        ├── admin.py
        ├── apps.py
        ├── __init__.py
        ├── migrations
        │   └── __init__.py
        ├── models.py
        ├── tests.py
        └── views.py

Then add this new app to the installed apps in settings.py:

INSTALLED_APPS = [
    ...
    'user_profile',
]

And then route the main page from the main app to our new user_profile app. This is done in the urls.py file in the main app:

from django.urls import path, include
from django.contrib import admin

urlpatterns = [
    path('^admin/', admin.site.urls),
    path('', include('user_profile.urls')),
]

Create a urls.py in your user_profile folder as well, and add the following to it:

from django.urls import path
from user_profile import views

urlpatterns = [
    path('', views.index),
]

The index view is still missing. Create it in views.py:

from django.shortcuts import render

def index(request):
    context = {}
    return render(request, 'user_profile/index.dtl', context)

Now we only need to create the Django template in the folder templates/user_profile/index.dtl (You can also use html as file extension if you don't get syntax highlighting for dtl files).

<!DOCTYPE html>
<html>
<body>
  <h1>My first Django OAuth app</h1>
</body>
</html>

The current file structure will look something like this:

[my-first-django-oauth-app]$ tree
.
└── src
    ├── db.sqlite3
    ├── manage.py
    ├── oauth_app
    │   ├── __init__.py
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    └── user_profile
        ├── admin.py
        ├── apps.py
        ├── __init__.py
        ├── migrations
        │   └── __init__.py
        ├── models.py
        ├── templates
        │   └── user_profile
        │       └── index.dtl
        ├── tests.py
        ├── urls.py
        └── views.py

Now you can start Django's built in development webserver:

$ python manage.py runserver 127.0.0.1:8080

This will show you the template you just created. Note that we are running on port 8080 because port 8000 is used on some systems. In order to stay consistent we will stick with port 8080 for local development. Because the port is also part of the OAuth callback URL, we will reduce the number of consumers we need to register for development.

Adding OAuth

Starting with version 1.2 the Python package social-core has a Mediawiki backend that works for any of Wikimedia Foundations wikis, but also for any other Mediawiki installation that has the OAuth Extension enabled.

Install the package for your virtual environment:

$ pip install social-auth-app-django

Add the following in your main app's settings.py:

INSTALLED_APPS = [
    ...
    'social_django',
]

MIDDLEWARE = [
    ...
    'social_django.middleware.SocialAuthExceptionMiddleware',
]

TEMPLATES = [
    {
        ...
        'OPTIONS': {
            'context_processors': [
                ...
                'social_django.context_processors.backends',
                'social_django.context_processors.login_redirect',
            ],
        },
    },
]

AUTHENTICATION_BACKENDS = (
    'social_core.backends.mediawiki.MediaWiki',
    'django.contrib.auth.backends.ModelBackend',
)

Then we need to add settings for the OAuth provider. You can register your application, after reading the OAuth for developers documentation.

We will register a consumer for development at this moment, so we will use the following settings:

  • Application name: use a name that indicates that you are developing locally
  • Leave "This consumer is for use only by <your username>" unchecked
  • Contact email address: Use a valid email where you can be reached.
  • Applicable project: All is fine
  • OAuth "callback" URL: http://127.0.0.1:8080/
  • Select: Allow consumer to specify a callback in requests and use "callback" URL above as a required prefix.
  • Types of grants being requested: Choose "User identity verification only, no ability to read pages or act on a user's behalf."
  • Public RSA key: You can leave this empty at the moment.

You should get back a response something like:

Your OAuth consumer request has been received.
You have been assigned a consumer token of xxxxxxxxxxxxxxxxxxxxxxxx
and a secret token of xxxxxxxxxxxxxxxxxxxxxxxx.
Be sure not to commit these keys in version control and keep them secret.

The consumer token is your SOCIAL_AUTH_MEDIAWIKI_KEY, and the secret token is your SOCIAL_AUTH_MEDIAWIKI_SECRET, in settings.py:

SOCIAL_AUTH_MEDIAWIKI_KEY = 'xxxxxxxxxxxxxxxxxxxxxxxx'
SOCIAL_AUTH_MEDIAWIKI_SECRET = 'xxxxxxxxxxxxxxxxxxxxxxxx'
SOCIAL_AUTH_MEDIAWIKI_URL = 'https://meta.wikimedia.org/w/index.php'
SOCIAL_AUTH_MEDIAWIKI_CALLBACK = 'http://127.0.0.1:8080/oauth/complete/mediawiki/'

N.B. The trailing '/' at the end of the callback is important. Login will silently fail without it.

After that, you need to apply the migrations in order to create the user models in the database.

$ python manage.py migrate

Next we need to up a few things for the profile view and the login. Update your user_profile apps urls.py to say something like this:

from django.urls import path, include
from user_profile import views

urlpatterns = [
    path('profile', views.profile, name='profile'),
    path('accounts/login', views.login_oauth, name='login'),
    path('oauth/', include('social_django.urls', namespace='social')),
    path('', views.index),
]

In settings.py we need to add the URL pattern named "login" as the LOGIN_URL and the URL pattern "profile" for LOGIN_REDIRECT_URL:

LOGIN_URL = 'login'
LOGIN_REDIRECT_URL = 'profile'

Create two new views for the profile and the login:

from django.shortcuts import render
from django.contrib.auth.decorators import login_required


def index(request):
    context = {}
    return render(request, 'user_profile/index.dtl', context)

@login_required()
def profile(request):
    context = {}
    return render(request, 'user_profile/profile.dtl', context)

def login_oauth(request):
    context = {}
    return render(request, 'user_profile/login.dtl', context)

Then create the login.dtl file:

<!DOCTYPE html>
<html>
<body>
  <h1>Login</h1>
  <a href="{% url 'social:begin' 'mediawiki' %}">Login with Wikimedia</a>
</body>
</html>

And the profile.dtl file:

<!DOCTYPE html>
<html>
<body>
  <h1>Profile</h1>
  {{ user }}
</body>
</html>

And finally update your index.dtl to say:

<!DOCTYPE html>
<html>
<body>
  <h1>My first Django OAuth app</h1>
  <a href="{% url 'profile' %}">Profile</a>
</body>
</html>

When you now restart the server and click on the "Profile" link you will be asked to give permission to your newly created OAuth consumer. After approving the consumer you will be redirected to the user page which will print the Wikimedia user name.

Deploying to Wikimedia Toolforge

After extensive testing you might want to share your new tool with a wider audience. If the tool fulfills the criteria of Toolforge you can host it there.

You can create a new tool account on toolsadmin.

Configure project for production environment

In order to make your project work in both environments and keep your secret keys out of source control you can for example define the following variables in venv-my-first-django-oauth-app/bin/activate. That means the keys will be available only if the environment is active:

...
export django_secret="very-secret-key"
export mediawiki_key="very-secret-mediawiki-key"
export mediawiki_secret="very-secret-mediawiki-secret"
export mediawiki_callback="http://127.0.0.1:8080/oauth/complete/mediawiki/"

And after reactivating the venv you will be able to access the variables in settings.py

SECRET_KEY = os.environ.get('django_secret')
...
SOCIAL_AUTH_MEDIAWIKI_KEY = os.environ.get('mediawiki_key')
SOCIAL_AUTH_MEDIAWIKI_SECRET = os.environ.get('mediawiki_secret')
SOCIAL_AUTH_MEDIAWIKI_URL = 'https://meta.wikimedia.org/w/index.php'
SOCIAL_AUTH_MEDIAWIKI_CALLBACK = os.environ.get('mediawiki_callback')

Now you can safely add our project to git. Run this in your project directory:

$ git init

You can then also add your requirements to your project:

$ pip freeze > requirements.txt

We also need to update the allowed hosts (See: Allowed-hosts) in settings.py:

ALLOWED_HOSTS = [
    'toolforge.org',
]

Don't forget to create a .gitignore file that excludes all files that should not be under source control. For example:

*.pyc
__pycache__/
*.sqlite3

Then add all the files to git. Make sure that only source files and no temporary files, or parts of your venv are staged. After that, you can make your first commit.

Use UWSGI locally

On Toolforge your Django app will run with UWSGI. In order to have a similar setup on the development machine we will create a similar setup using Nginx. First we need to create an app.ini in our src directory:

[uwsgi]
module = oauth_app.wsgi

plugins = python3

chdir = /home/YOURUSERNAME/my-first-django-oauth-app/src
home = /home/YOURUSERNAME/venv-my-first-django-oauth-app

master = true
processes = 5

uid = YOURUSERNAME
socket = /run/uwsgi/django-oauth.sock
chown-socket = YOURUSERNAME:nginx
chmod-socket = 664
vacuum = true

die-on-term = true

You should be able to start the uwsgi file on its own by using: uwsgi app.ini. If it fails this is often a permission error with the /run/uwsgi directory. The current user needs to be able to create the socket file in this directory.

For this file we will create a service in /etc/systemd/system/django-oauth.uwsgi.service:

[Unit]
Description=uWSGI instance to serve django oauth app

[Service]
ExecStartPre=-/usr/bin/bash -c 'mkdir -p /run/uwsgi; chown YOURUSERNAME:nginx /run/uwsgi'
ExecStart=/usr/bin/bash -c 'cd /home/YOURUSERNAME/my-first-django-oauth-app/src; source /home/YOURUSERNAME/venv-my-first-django-oauth-app/bin/activate; uwsgi --ini app.ini'

[Install]
WantedBy=multi-user.target

Ubuntu for example does not have /usr/bin/bash and you need to use /bin/bash. Errors like this can be seen when using sudo systemctl status django-oauth.uwsgi.service.

You can then start this service by running sudo systemctl start django-oauth.uwsgi.service. If that has worked you can continue to configure the webserver. Go to /etc/nginx/nginx.conf and add the following configuration:

http {

    ...

    server {
        listen 8080;
        server_name 127.0.0.1;

        location / {
            include uwsgi_params;
            uwsgi_pass unix:/run/uwsgi/django-oauth.sock;
        }
    }

    ...

    server { ...

After that you can either restart or start the nginx server (sudo systemctl start nginx). You can check if everything is working correctly by entering: sudo systemctl status nginx. On an SELinux enabled system you might still need to set the correct target context for the sock file. Now go to 127.0.0.1:8080 and see if your page still loads correctly.

Login and deploy

After configuring two-factor authentication you can log into Toolforge using:

$ ssh YOUR_USER_NAME@login.toolforge.org

After the successful login you need to switch to your new tool:

$ become YOUR_TOOL_NAME

You can use the following commands to control the web server:

$ webservice --backend=kubernetes python status/start/stop/restart

Create the following file structure for the web server config file:

.
└── www
    └── python
        ├──uwsgi.ini
        └──src

In the uwsgi.ini file add the following:

[uwsgi]
check-static = /data/project/YOUR_TOOL_NAME/www/python/static

Any files in this static directory will be publicly accessible on the internet.

In the python directory you can clone your git repository (The dot at the end clones into your current directory instead of creating another subfolder):

[python] $ git clone ADDRESS_OF_REPOSITORY .

You also have to create a virtual environment on the server using:

[python] $ virtualenv -p /usr/bin/python3.4 venv

Then you are ready to request your production keys (which you need to be even more careful about not committing):

  • Application name: use a name that indicates that this is production
  • Contact email address: Use a valid email where you can be reached.
  • Applicable project: All is fine
  • OAuth "callback" URL: https://YOUR-TOOL-NAME.toolforge.org/
  • Select: Allow consumer to specify a callback in requests and use "callback" URL above as a required prefix.
  • Types of grants being requested: Choose "User identity verification only, no ability to read pages or act on a user's behalf."
  • Public RSA key: You can leave this empty at the moment.

Make the virtual environment activate file non-world-readable to ensure that nobody else will be able to access the production keys in it:

[python] $ chmod go-rwx venv/bin/activate

Then edit the file to add your production keys to it:

export django_secret="very-secret-production-key"
export mediawiki_key="very-secret-mediawiki-production-key"
export mediawiki_secret="very-secret-mediawiki-production-secret"
export mediawiki_callback="https://YOUR-TOOL-NAME.toolforge.org/oauth/complete/mediawiki/"

Activate the environment to install the dependencies and apply the migrations, then deactivate it again:

[python] $ source venv/bin/activate
[python] $ pip install -r requirements.txt

[python] $ python src/manage.py makemigrations
[python] $ python src/manage.py migrate

[python] $ deactivate

Running the 'migrate' command will also create the database as a sqlite file. The file with our current settings is at /www/python/src/db.sqlite3. You should take care to back up this file and be confidential with the information that is stored within it.

With the current setup you also have to create a 2nd app.py in ~/www/python/src where the standard webservice will look for it. It also expects the variable to be named app:

import os

from django.core.wsgi import get_wsgi_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "oauth_app.settings")

app = get_wsgi_application()

After making these configurations you should be able to start the webserver using:

$ webservice --backend=kubernetes python start

In case there are any problems with the configuration they will appear in the tools root directory in the file: ~/uwsgi.log.

The project structure as of now will look like this:

$ tree
.
├── logs
├── replica.my.cnf
├── service.manifest
├── uwsgi.log
└── www
    └── python
        ├── requirements.txt
        ├── src
        │   ├── app.py
        │   ├── db.sqlite3
        │   ├── manage.py
        │   ├── oauth_app
        │   │   ├── __init__.py
        │   │   ├── settings.py
        │   │   ├── urls.py
        │   │   └── wsgi.py
        │   └── user_profile
        │       ├── admin.py
        │       ├── apps.py
        │       ├── __init__.py
        │       ├── migrations
        │       │   └── __init__.py
        │       ├── models.py
        │       ├── templates
        │       │   └── user_profile
        │       │       ├── index.dtl
        │       │       ├── login.dtl
        │       │       └── profile.dtl
        │       ├── tests.py
        │       ├── urls.py
        │       └── views.py
        ├── uwsgi.ini
        └── venv

You can view a working example of the OAuth app at:

https://my-first-django-oauth-app.toolforge.org/

Scale up your database

So far the tutorial only used sqlite as a database. This is fine for demo applications, but will quickly result in performance or scaling issues with any operations that require reads or writes to the database. It will also strain the Toolforge servers, which is something we also want to avoid. You can use user databases on the tools.db.svc or create your own database server on Cloud VPS.

Links

Tutorials

Code

Further Reading

Other tools

Communication and support

We communicate and provide support through several primary channels. Please reach out with questions and to join the conversation.

Communicate with us
Connect Best for
Phabricator Workboard #Cloud-Services Task tracking and bug reporting
IRC Channel #wikimedia-cloud connect
Telegram bridge
mattermost bridge
General discussion and support
Mailing List cloud@ Information about ongoing initiatives, general discussion and support
Announcement emails cloud-announce@ Information about critical changes (all messages mirrored to cloud@)
News wiki page News Information about major near-term plans
Cloud Services Blog Clouds & Unicorns Learning more details about some of our work
Wikimedia Technical Blog techblog.wikimedia.org News and stories from the Wikimedia technical movement