You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Analytics/Systems/Superset: Difference between revisions
(Add predefined sum-metric usage note)
|Line 22:||Line 22:|
=== Account Creation ===
=== Account Creation ===
a user superset. in the , and .
=== Upgrading ===
=== Upgrading ===
Revision as of 20:55, 7 January 2020
Superset is an Apache incubator project, originally started at AirBnB. It enables visualizations and dashboards built from various analytics data sources. WMF's Superset instance can be found at https://superset.wikimedia.org. Like Turnilo, it provides access to various Druid tables.
Before requesting access, please make sure you:
- have a functioning Wikitech login. Get one: https://toolsadmin.wikimedia.org/register/
- are an employee or contractor with wmf OR have signed an NDA
Depending on the above, you can request to be added to the wmf group or the nda group. Please indicate the motivation on the task about why you need access and ping the analytics team if you don't hear any feedback soon from the Opsen on duty.
Once you are in either the wmf or nda LDAP groups,
Superset will automatically create an account for you. Just visit https://superset.wikimedia.org. (NOTE: As of September 24, 2018, there is a bug with Superset that means that user accounts are not automatically created. This bug manifests as a stack trace with the message "AttributeError: 'bool' object has no attribute 'login_count'" when attempting to access Superset. If you see this error, request that the Analytics team create your user for you.)
- The "Druid Datasources" list shows ingested tables that are available for querying. As of October 2018, this includes e.g. daily and hourly pageviews data (the daily version is only updates once a month, but goes further back), a sampled excerpt of webrequest data, unique devices, and a few select EventLogging schemas. If a recently created Druid datasource is not yet visible in the list, try clicking "Scan New Datasouces".
- NULL values don't show up properly in the values selection dropdown list for filters (i.e. one can't use that dropdown to exclude NULL values from a chart or limit it to NULL values). But one can use the regex option instead: Type in ".+" (without the quotes), and accept the offer to create that as an option.
- By default, always use predefined SUM metrics when available. When choosing a metric then picking the SUM aggregation function, the aggregation is managed by superset and uses the
floatSumoperator. This operator uses 32 bits floats instead of 64 bits longs or double, leading to inaccuracies. Usually predefined
SUM(...)metrics are available and should be used, as they are manually defined using
longSum64 bits operators.
Account creation should be handled automatically the first time a user logs into superset. They need to be in the wmf or nda LDAP group, and should log in using their shell username and LDAP password.
To upgrade, first follow the instructions in the analytics/superset/deploy README to update the deploy repository. Once deployed, activate the superset virtualenv, add /etc/superset to PYTHONPATH (to allow superset to pick up configuration) and follow the Superset upgrade instructions (minus the pip install superset --upgrade part). This should be something like:
. /srv/deployment/analytics/superset/venv/bin/activate export PYTHONPATH=/etc/superset superset db upgrade superset init
This assumes that one has already filed the change for the superset deploy repository. The first thing to do is test the change on the staging instance, an-tool1005.eqiad.wmnet:
# ssh to deploy1001 and set the working directory ssh deploy1001.eqiad.wmnet cd /srv/deployment/analytics/superset/deploy # create a new branch from the master one, name it as you prefer git checkout -B testing_something_important # cherry pick the change in the new branch created git cherry-pick $change-from-gerrit # deploy only to an-tool1005, without logging in the ops's sal scap deploy --no-log-message -f -l an-tool1005.eqiad.wmnet "Test deployment for something important"
Then check if Superset works as expected:
# Create a ssh tunnel and then test it via localhost:8080 on the browser ssh -L 8080:an-tool1005.eqiad.wmnet:80 an-tool1005.eqiad.wmnet
If you are happy with the Superset version, then merge and deploy to the production host:
# ssh to deploy1001 and set the working directory ssh deploy1001.eqiad.wmnet cd /srv/deployment/analytics/superset/deploy scap deploy -l analytics-tool1004.eqiad.wmnet "Deployment for something important"
systemctl status superset.service
systemctl restart superset
- https://github.com/wikimedia/incubator-superset Wikimedia fork of Superset