Proton

Proton is a service that converts the Wikipedia articles into PDF. It uses Pupeeteer to fetch the Wikipedia page, render it in headless chromium, and then calls the puppeteer page.pdf() call to return PDF version of the article.

Monitoring

• Kibana dashboard
• Grafana dashboard for PDF metrics
• Prometheus breakdown for the Proton cluster on codfw
• Prometheus breakdown for the Proton cluster on eqiad
• Jenkins Job Builder docs for updating Jenkins jobs
• Icinga uses the swagger file and performs HTTP checks for Desktop/Mobile prints plus it verifies the requesting non-existing page.

Deploying changes

Proton is deployed using scap3. Doing deployments with scap3 is very easy. You just run scap deploy, which pushes the new state to all backends and restarts them. You should have deploy access and be a member of the proton-admins group puppet group.

Pre-deploy checks

None

Prepare the deploy patch

Proton is based on Services template. For more detailed information please refer to [Services/Deployment]

• Create a short deployment summary on mw:Proton/Deployments from git log --cherry-pick {from}...{to}. Don't include all commits, but only notable fixes and changes (ignore rt-test fixes, code cleanup updates, test updates, etc). (The above command will do the right thing if {from} was on a branch and had patches cherry-picked from {to}, although if there were conflicts during the cherry-pick to {from} the patch will still appear in the log for {to}.)
• Prepare a chromium-render/deploy deploy repo commit and push for +2

Verify deployment version on beta after the deploy patch is merged

• Deploy code (if not already there) to the beta cluster (same instructions as below but ssh to deployment-chromium01.deployment-prep.eqiad.wmflabs)

Be around on IRC

• Add yourself to the "deployer" field of Deployments if you're not already there
• Be online in freenode #wikimedia-operations (and stay online through the deployment window)

Deploying the latest version of Proton

Now to do the deploy:

ssh deployment.eqiad.wmnet
tin$ cd /srv/deployment/proton/deploy
tin$ git pull
tin$ git submodule update --init
tin$ scap deploy 'Updating Proton to <new hash> (T<bug number>, T<bug number>)'

Scap will log the completion of the deploy, but if you want to add additional information to the SAL, make a comment in #wikimedia-operations with something like

Post-deploy checks

• Verify the Grafana dashboard for PDF metrics that service handles similar number of requests
• Verify that logstash has no new errors
• Use the Proton testing tool and verify that generated PDF is still correct

Restarting

In case you need to restart Proton without any deployments (for example, to reload mediawiki configs from config or other deployments),

• Restart proton hosts, from deployment.eqiad.wmnet (production) or deployment-chromium01.deployment-prep.eqiad.wmflabs (beta)

  cd /srv/deployment/proton/deploy && scap deploy --service-restart

When something goes wrong

Reverting a Proton deployment

Code

ssh deployment.eqiad.wmnet
cd /srv/deployment/proton/deploy
scap deploy --rev <sha>

Data flow