Data Engineering/Systems/Superset/Administration

Upgrades

To upgrade, first follow the instructions in the analytics/superset/deploy README to update the deploy repository. Once deployed, activate the superset virtualenv, add /etc/superset to PYTHONPATH (to allow superset to pick up configuration) and follow the Superset upgrade instructions (minus the pip install superset --upgrade part). This should be something like:

sudo su superset
source /srv/deployment/analytics/superset/venv/bin/activate
export PYTHONPATH=/etc/superset
export FLASK_APP=superset
superset db upgrade
superset init

Deploments

Deploy to staging

This assumes you have a change open for the analytics/superset/deploy repository.

# ssh to deploy1002 and set the working directory
ssh deploy1002.eqiad.wmnet
cd /srv/deployment/analytics/superset/deploy

# Checkout your change from gerrit (use "Download patch" in the top-right menu)
git fetch https://gerrit.wikimedia.org/r/analytics/superset/deploy refs/changes/83/789683/1 && git checkout FETCH_HEAD

# deploy only to an-tool1005, without logging in the ops's sal
scap deploy --no-log-message -f -l an-tool1005.eqiad.wmnet "Test deployment for something important"

Sync the staging database with the production one

When testing a new release in the staging environment it is nice to get the same dashboards as in production, since two different databases are used and they get out of sync very quickly (nobody updates dashboards in staging). The procedure is the following:

# The two databases live on the same mariadb instance
ssh an-coord1001.eqiad.wmnet
# Moving to /srv since it is on a separate partition with more free space
cd /srv
# Dump the production db
sudo sh -c 'mysqldump superset_production > superset_production_$(date +%s).sql'
# Get the filename of the production dump
ls -l
# Connect to mysql and drop the staging db
sudo mysql
# Drop and re-create the staging database
drop database superset_staging;
create database superset_staging DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
exit

# Load the production database into the staging one. Change the filename according
# to the dump previously taken!
sudo mysql superset_staging < superset_production_111111.sql

# Then remember that some specific production things need to be changed once superset
# runs. For example, in databases -> presto-analytics (in the superset ui) the kerberos principal
# used is saved in the config and changes between production and staging (the hostname is different).

Test as different user on staging

In order to test features such as permissions, you may want to impersonate a different user.

First, add an extension to your browser that allows you to set custom headers, such as https://addons.mozilla.org/en-US/firefox/addon/modify-header-value/ for Firefox.

Then, set the following headers for localhost:8080:

X-Cas-Uid: <user you want to impersonate>
X-Forwarded-Proto: http
Host: superset-next.wikimedia.org

Connect directly to the superset server by opening an SSH tunnel on an-tool1005 port 9080:

ssh -NL 8080:an-tool1005.eqiad.wmnet:9080 an-tool1005.eqiad.wmnet

Open http://localhost:8080 in your browser, and you should be logged in as the user specified by X-Remote-User. You can see who you are logged in as by viewing your profile from the Settings menu.

If you change the header to be a different user, logout via the Settings menu to change to be the new user.

Deploy to production

# ssh to deployment.eqiad.wmnet and set the working directory
ssh deployment.eqiad.wmnet
razzi@deploy1002:~$ cd /srv/deployment/analytics/superset/deploy
$ scap deploy -l an-tool1010.eqiad.wmnet "Deployment for something important"

Don't forget to run migrations after you deploy (see Analytics/Systems/Superset#Upgrading).

Test a different role's permissions

During a couple of superset upgrades, there have been permissions removed from the Alpha role, which causes our users to see errors, sometimes, frustratingly, "unknown error" (at least by looking in the network tab you can see 401 unauthorized status).

It's theoretically possible to create test users, I don't have a method to do so currently so instead I modify my user to have the role I want to test. I do so in the superset app shell; other database operations could be done manually this way.

Log in to the superset instance (in this example I am connected to production, an-tool1010), switch to the superset user, set up the python environment, and start the shell:

razzi@an-tool1010:~$ sudo su superset
superset@an-tool1010:/home/razzi$ source /srv/deployment/analytics/superset/venv/bin/activate
(venv) superset@an-tool1010:/home/razzi$ export PYTHONPATH=/etc/superset
(venv) superset@an-tool1010:/home/razzi$ superset shell
Loaded your LOCAL configuration at [/etc/superset/superset_config.py] ...
>>> app
<SupersetApp 'superset.app'>

The database connection is accessible under app.extensions['sqlalchemy'].db.session:

>>> session = app.extensions['sqlalchemy'].db.session
>>> from flask_appbuilder.security.sqla.models import User
>>> me = session.query(User).filter_by(username='razzi').first()
>>> me
razzi -
>>> me.roles
[Alpha]
# You can modify roles as follows:
>>> Role = me.roles[0].__class__
>>> session.query(Role).all()
[Admin, Alpha, Gamma, granter, Public, sql_lab]
>>> [admin, alpha, *roles] = session.query(Role).all()
>>> me.roles.append(alpha)
>>> me.roles.remove(admin)
>>> session.add(me)
>>> session.commit()

At this point I'd have alpha and not admin roles (this can be confirmed by going to your superset profile) and I can test what other Alpha users see.

How to

See status

systemctl status superset.service

Restart

systemctl restart superset