Tool:XTools

**Toolforge tools**
XTools
Website	https://xtools.wmflabs.org/
Description	Suite of tools to analyze user and page data on WMF wikis
Keywords	xtools, statistics, analytics
Author(s)	Matthewrbowker, MusikAnimal, Samwilson; adapted from older version by X! and Hedonil
Maintainer(s)	Matthewrbowker, MusikAnimal, Samwilson (View all)
Source code	GitHub (Mirrored on Phabricator as rXTR)
License	GNU General Public License 3.0 or later
Issues	Open tasks · Report a bug
Admin log	Nova Resource:Tools.xtools/SAL; Nova Resource:Tools.xtools-dev/SAL

This page is for documentation relating to the WMF installations of XTools at xtools.wmflabs.org (production) and xtools-dev.wmflabs.org (staging). For general installation, configuration, and development of the XTools software, please see xtools.readthedocs.io.

There are three instances currently configured, one for staging, one for the main production app server, and one for the API (view details in the Openstack browser). The prod instances relate to the Toolforge account xtools, and the staging instance relates to xtools-dev; these are where the matching database users come from, and where we send maintainers' emails.

Note to maintainers: we don't backup any server configuration, so please document everything here (until task T170514 is resolved).

Contact

The maintainers can be emailed at tools.xtoolstools.wmflabs.org (note that this means that the maintainers of three separate things need to be kept in sync: the VPS account and the two Toolforge accounts).

Production

Production XTools is hosted on on a Wikimedia VPS instance. To log into the server, make sure you've been added as a maintainer to the xtools project. Then set up SSH and connect via ssh xtools-prod01.xtools.eqiad.wmflabs and go to the /var/www directory. Not quite everything in this directory is in the Git repository.

Logs are written to /var/www/var/logs/prod.log, but only during a request where an error or high-priority log entry was made. This is why you'll see debug-level log entries in prod.log. You might also need to check /var/log/apache2/error.log for Apache-level errors.

Web access stats are available at https://xtools.wmflabs.org/awstats/awstats.pl

OAuth consumer: XTools 1.0

Database is s51187__xtools_prod on tools.labsdb, we're connecting as user s51187 (which is the same as the old XTools database user).

Web server configuration is all in /etc/apache2/sites-available/xtools.conf.

There's a /var/www/deploy.sh script that runs every 10 minutes (from www-data's crontab) and if required updates to the latest release. The output of this is mailed to the maintainers.

There is also a dedicated API server, which lives at xtools-prod02.xtools.eqiad.wmflabs. All requests to https://xtools.wmflabs.org/api go to this server.

Building a new instance

First create a new instance running on debian-jessie. Any production node should be at least a m1.medium with 4 GB of RAM, but for the main app server a m1.large is probably best.

Once the instance has been spawned, SSH in and follow these steps:

Install PHP along with some dependencies (from https://php.net/manual/en/install.unix.debian.php).

sudo apt-get install php5-common libapache2-mod-php5 php5-cli
sudo apt-get install php5-mysql php5-curl php5-intl

Install Node and npm.

sudo apt-get install nodejs
sudo apt-get install npm

Symlink nodejs to node, so the uglify-js plugin runs the right command.
```
sudo ln -s /usr/bin/nodejs /usr/bin/node
```
Install composer by following these instructions, but make sure to install to the /usr/local/bin directory and with the filename composer, e.g.:
```
sudo php composer-setup.php --install-dir=/usr/local/bin --filename=composer
```

Clone the repository, first removing the html directory created by Apache.

cd /var/www && sudo rm -rf html
sudo git clone https://github.com/x-tools/xtools.git .

Run composer install and fill in the necessary parameters, using https://xtools.readthedocs.io/en/latest/configuration.html as a guide. For most options you can use the default.

Create the deploy script at /var/www/deploy.sh with the following:

#!/bin/bash

cd /var/www

git fetch --quiet origin 2>&1

## Find the highest and current tags
HIGHEST_TAG=$(git tag | sort --version-sort | tail --lines 1)
CURRENT_TAG=$(git describe --tags)

## Exit and say nothing if we're already at the highest tag.
if [[ $CURRENT_TAG == $HIGHEST_TAG ]]; then
    # The following line can be temporarily uncommented as-needed
    # to force www-data to clear the production cache:
    # ./bin/console cache:clear --env prod
    exit 0
fi

## If there's an update, pull and install it.
git checkout $HIGHEST_TAG
export SYMFONY_ENV=prod
/usr/local/bin/composer install --no-dev --optimize-autoloader
./bin/console cache:clear --env prod
./bin/console doctrine:migrations:migrate --env prod --no-interaction
./bin/console assetic:dump --env prod

Make sure the deploy script is executable, and that all the files in the repo are owned by www-data.
```
sudo chmod 744 deploy.sh
sudo chown -R www-data:www-data .
```

Create the web server configuration file at /etc/apache2/sites-available/xtools.conf with the following:

<VirtualHost *:80>
        DocumentRoot /var/www/web
        ServerName xtools.wmflabs.org

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        <Directory /var/www/web/>
             Options Indexes FollowSymLinks
             AllowOverride All
             Require all granted
        </Directory>

        # Deny access to known spiders.
        SetEnvIfNoCase User-Agent "(Baiduspider|CCBot|scrapy\.org|kinshoobot|YisouSpider|Sogou web spider|yandex\.com/bots|TweetmemeBot|SeznamBot|datasift\.com/bot|Googlebot|Yahoo! Slurp|Python-urllib|BehloolBot|MJ12bot)" bad_bot

        <Directory /var/www/>
                Options Indexes FollowSymLinks
                AllowOverride None
                Require all granted
                Deny from env=bad_bot
        </Directory>

        Alias /awstatsclasses "/usr/share/awstats/lib/"
        Alias /awstats-icon/ "/usr/share/awstats/icon/"
        Alias /awstatscss "/usr/share/doc/awstats/examples/css"
        ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
        ScriptAlias /awstats/ /usr/lib/cgi-bin/
        <Directory /usr/lib/cgi-bin/>
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
                Require all granted
        </Directory>

        SetEnvIf Request_URI "^/robots\.txt$" dontlog
        
        RewriteEngine On
        RewriteCond %{HTTP:X-Forwarded-Proto} !https
        RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R=301,L]

</VirtualHost>

Enable the mod-rewrite Apache module, and enable the web server configuration.
```
sudo a2enmod rewrite
sudo a2ensite xtools
sudo service apache2 reload
```
Setup the crontab to run the deploy script every 10 minutes. We'll do this under the www-data user.
```
sudo su root
sudo crontab -e -u www-data
```
Then add this to the bottom of the crontab:
```
MAILTO=tools.xtools@tools.wmflabs.org
*/10 * * * * /var/www/deploy.sh
```
Wait for the email indicating composer ran successfully, assets were dumped, etc. If all goes well, you need only to gracefully (re)start apache, which is how you should always restart apache on production:
```
sudo apachectl -k graceful
```

Setting up an API server

The API server itself can be built the same as the app server, except you'll need to update the ServerName in xtools.conf with the correct domain. Beyond that, we just need to configure proxy settings on the main app server so that all requests to /api go to the API server. You can this by following these steps:

Install libapache2-mod-proxy-html and libxml2-dev

sudo apt-get install libapache2-mod-proxy-html libxml2-dev

Enable the necessary modules (if some are already enabled it will simply make sure they are active):

sudo a2enmod proxy
sudo a2enmod proxy_http
sudo a2enmod proxy_ajp
sudo a2enmod rewrite
sudo a2enmod deflate
sudo a2enmod headers
sudo a2enmod proxy_balancer
sudo a2enmod proxy_connect
sudo a2enmod proxy_html
sudo a2enmod xml2enc

And in /etc/apache2/sites-available/xtools.conf, within the <VirtualHost> block, add this to the bottom:
```
ProxyPreserveHost On
ProxyPass /api http://X.X.X.X:80/app.php/api
ProxyPassReverse /api http://X.X.X.X:80/app.php/api
```
...replacing X.X.X.X with the IP of the API server.
Finally, restart apache with sudo apachectl -k graceful

Staging

SSH to xtools-dev02.xtools.eqiad.wmflabs (see notes above in #Production about getting access).

Database is s53003__xtools_dev on tools.labsdb, we're connecting as user s53003.

OAuth consumer: xtools-dev 1.0

The code on the staging server is kept up to date with the master branch with the following /var/www/deploy.sh script (run every 10 minutes from www-data's crontab):

#!/bin/bash

export SYMFONY_ENV=prod
cd /var/www

## See if there's any update.
GITFETCH=$(git fetch && git diff origin/api-key 2>&1)
if [ -z "$GITFETCH" ]; then
        exit 0
fi

## If there's an update, pull and install it.
git checkout master
git pull origin master
/usr/local/bin/composer install --no-dev --optimize-autoloader
./bin/console cache:clear --env prod
./bin/console doctrine:migrations:migrate --env prod --no-interaction
./bin/console assetic:dump --env prod

Setting up the staging server is the same as production, except you would use a smaller box (m1.small), and use the above deploy script instead of the one that goes off of tags. You also need to update the ServerName in xtools.conf accordingly.

Killing slow queries

Both servers run slow-query killing scripts for their respective database users.

Slow queries are killed with pt-kill by this daemon (see /var/www/kill-slow-queries.sh):

/usr/local/bin/pt-kill --user=xxxx --password=xxxx --host=enwiki.labsdb \
    --busy-time=90 \
    --log /var/www/web/killed_slow_queries.txt \
    --match-info "^(select|SELECT|Select)" \
    --kill --print --daemonize --verbose

The daemon can be stopped at any time by creating /tmp/pt-kill-sentinel.

The killed queries are publicly logged for both production and staging.

These logs are kept to a maximum of 2 1 kb files via /etc/logrotate.d/xtools:

/var/www/web/killed_slow_queries.txt {
    size 1k
    rotate 2
    copytruncate
}

Note also that pt-kill is installed manually (it's just a single Perl file),

and that it requires libdbi-perl and libdbd-mysql-perl.

Installation on Toolforge

Old XTools is still installed on Toolforge at https://tools.wmflabs.org/xtools and new XTools was previously installed at https://tools.wmflabs.org/xtools-dev — the instructions here are for the latter.

To create a XTools project on Toolforge:

git clone https://github.com/x-tools/xtools.git
composer install
You will be asked a series of questions. See https://xtools.readthedocs.io/en/latest/configuration.html for documentation. For Toolforge:
- wiki_url – leave blank
- app.is_labs = 1
- app.single_wiki = 0
- app.load_stylesheets_from_cdn = 0
- app.replag_threshold = 30
- Enable the tool that you are setting up, and set the rest to 0
You can use the app/config/table_map.yml file to force the app to use certain tables in place of the defaults. For instance on Toolforge, you'll want to map revision to revision_userindex since it will be faster.

Set the .lighttpd.conf file to:

url.rewrite-if-not-file += (
  "(.*)" => "/xtools-dev/app.php/$0",
  "^/$" => "$0"
)

Start the webservice with webservice --backend=kubernetes start

XTools

Website	https://xtools.wmflabs.org/
Description	Suite of tools to analyze user and page data on WMF wikis
Keywords	xtools, statistics, analytics
Author(s)	Matthewrbowker, MusikAnimal, Samwilson adapted from older version by X! and Hedonil
Maintainer(s)	Matthewrbowker, MusikAnimal, Samwilson (View all)
Source code	GitHub (Mirrored on Phabricator as rXTR)
License	GNU General Public License 3.0 or later
Issues	Open tasks · Report a bug
Admin log	Nova Resource:Tools.xtools/SAL Nova Resource:Tools.xtools-dev/SAL

Tool:XTools

Contents

Contact

Production

Building a new instance

Setting up an API server

Staging

Killing slow queries

Installation on Toolforge

See also

Navigation menu

Tool:XTools

Contact

Production

Building a new instance

Setting up an API server

Staging

Killing slow queries

Installation on Toolforge

See also

Navigation menu

Search