You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Wikidata Query Service/Wikimedia Commons Query Service/Runbook
Wikimedia Commons Query Service is currently in beta and is served from an instance in Horizon. Instance name is wcqs-beta-01 (full hostname - wcqs-beta-01.wikidata-query.eqiad.wmflabs). Any puppet configuration is served from puppet master on wdqspuppet (wdqspuppet.wikidata-query.eqiad.wmflabs). Service uses OAuth 1.1a authentication, that is handled by nginx reverse proxy. Service is bound to https://wcqs-beta.wmflabs.org/ endpoint.
OAuth 1.1a setup
Our current setup for authentication is based on OAuth 1.1a, provided by Wikimedia (commons.wikimedia.org based). The authorization itself is handled by nginx reverse proxy (auth code) and simple java application (code here). Java application is piggybacked with WCQS deployments of Blazegraph.
OAuth requires credentials to be provided. Since currently service is in beta, we cannot use standard methods of having secret credentials, that is being used in production systems, is unavailable. To handle that, we use wdqspuppet instance to host our own puppetmaster, with the credentials saved as a local commit. Puppetmaster's puppet repo needs to be updated manually, but the credentials commit should be rebased onto changed production branch. Puppet repo clone is located at /var/lib/git/operations/puppet/.
We don't have any deployment set up for, so it's being done manually. To do that, you need to log in to wcqs-beta-01 and execute following steps.
In the /srv/wdqs-package execute:
git pull git fat pull sudo systemctl restart wcqs-blazegraph
For potential updates to puppet, on wdqspuppet, in /var/lib/git/operations/puppet/:
sudo git fetch sudo git rebase origin/production
On wcqs-beta-01, execute:
sudo puppet agent -tv sudo systemctl restart wcqs-blazegraph
So far WCQS isn't updated real-time and it relies on a weekly process of reloading the data in zero-downtime manner - by indexing a standby namespace and switching it with active one after being done. Process is defined as a cronjob (here), but of course can be also done manually, by executing script wcqs-data-reload.sh in /srv/wdqs-package/ directory.
Query count summary
We have no easy way to push metrics out of the beta instance, but to provide at least simple quantity metrics we save the same metrics we normally send to event-gate, to storage. There is s script that summarizes them:
./summarizeEvents.sh -e <log event path, can be a prefix for rolling logs> -s <output csv>