You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Parsoid: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Subramanya Sastry
(Add lang attribute to the syntaxhighlight use)
imported>Cscott
(→‎Deploying the latest version of Parsoid: Use a canary from codfw, not eqiad)
Line 54: Line 54:
You will then get status updates. If any minions are not ok, then retry the deploy until all are. Proceed with 'y' when all minions are ok for each step.
You will then get status updates. If any minions are not ok, then retry the deploy until all are. Proceed with 'y' when all minions are ok for each step.


Nodes are not automatically restarted. First restart Parsoid on a canary (wtp1001 is used here as an example).
Nodes are not automatically restarted. First restart Parsoid on a canary (wtp2001 is used here as an example).


<source lang="bash">
<source lang="bash">
ssh wtp1001.eqiad.wmnet
ssh wtp2001.codfw.wmnet
wtp1001:~$ sudo service parsoid restart
wtp2001:~$ sudo service parsoid restart
parsoid stop/waiting
parsoid stop/waiting
parsoid start/running, process 25849
parsoid start/running, process 25849
wtp1001:~$
wtp2001:~$
</source>
</source>


Add a log entry<syntaxhighlight lang="bash">
Add a log entry<syntaxhighlight lang="bash">
  !log synced code; restarted Parsoid on <node-name> as a canary
  !log synced code; restarted Parsoid on <node-name> as a canary
</syntaxhighlight>Monitor [https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=cpu_report&c=Parsoid+eqiad&h=wtp1001.eqiad.wmnet&tab=m&vn=&hide-hf=false&mc=2&z=small&metric_group=ALLGROUPS wtp1001 on ganglia] for a while to make sure things seem okay. (FIXME: describe how to do so). You can <code>curl http://localhost:8000/_version</code> to verify Parsoid on wtp1001 is running correctly. It should output the expected sha. Keep an eye on Kibana for log entries from the node to make sure nothing untoward is happening.
</syntaxhighlight>Monitor [https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=cpu_report&c=Parsoid+codfw&h=wtp2001.codfw.wmnet&tab=m&vn=&hide-hf=false&mc=2&z=small&metric_group=ALLGROUPS wtp2001 on ganglia] for a while to make sure things seem okay. (FIXME: describe how to do so). You can <code>curl http://localhost:8000/_version</code> to verify Parsoid on wtp2001 is running correctly. It should output the expected sha. Keep an eye on Kibana for log entries from the node to make sure nothing untoward is happening.


Then to restart the rest of the nodes:<source lang="bash">
Then to restart the rest of the nodes:<source lang="bash">

Revision as of 21:39, 5 April 2016

Parsoid is a service that parses converts between wikitext and HTML. The HTML contains additional metadata that allows it to be converted back ("round-tripped") to wikitext.

  • VisualEditor fetches the HTML for a given page from Parsoid, edits it, then delivers the modified HTML to Parsoid, which converts it back to wikitext. Parsoid is a stateless HTTP server running on port 8000.
  • Flow (as configured on WMF wikis with $wgFlowContentFormat = 'html') works the other way around. When a user creates a post Flow uses Parsoid to convert the wikitext to HTML and Flow stores the HTML in ExternalStore. If someone later edits a post Flow uses Parsoid to convert the HTML back to wikitext for editing.

Monitoring

  • Parsoid eqiad cluster in Ganglia, only lists the worker machines.
  • Icinga has service checks for HTTP on port 8000 on both the individual backends and on the LVS service IP.
  • pybal does health checks on all backends every second, and depools boxes that are down as long as the % of depooled boxes does not exceed 50%. To see these health checks and depools/repools happen in real time, run ssh parsoid.svc.eqiad.wmnet (this will drop you into either lvs1003 or lvs1006, depending on which is active), then tail -f /var/log/pybal.log | grep parsoid
  • Logging happens in /var/log/parsoid/parsoid.log. There is a log rotation setup in /etc/logrotate.d/parsoid.

Deploying changes

Parsoid is deployed using git-deploy. Doing deployments with git-deploy is very easy . You run git deploy start, make whichever changes you need to make to the git clone (such as pulling, changing branches, committing live hacks, etc.), then run git deploy sync. The sync command pushes the new state to all backends and restarts them. You should have deploy access and be a member of the parsoid-admins puppet group.

Pre-deploy checks

Prepare the deploy patch

  • Check http://parsoid-tests.wikimedia.org/regressions/between/{from}/{to} where {from} is the last deployed hash from mw:Parsoid/Deployments and {to} is the latest tested commit (which we're about to deploy)
    • http://parsoid-tests.wikimedia.org/commits gives you a nice radio-button interface to create this URL
    • BEWARE: if you get the output total regressions between selected revisions: 0, it is extremely likely that you mistyped the hash or that we didn't actually run round-trip tests for that particular hash. (This is a bug, we should probably give a better message in this case.)
    • Since we are using current revision of titles in round-trip testing, edits to pages can show up as false regressions. tools/regression-testing.js is useful in filtering those out. Grab the wiki:title pairs that show up as regressions in the regressions url, save it to a file and feed it to the regressions tool and you will get a list of pages to look more closely, if necessary.
  • Create a short deployment summary on mw:Parsoid/Deployments from git log {from}..{to}. Don't include all commits, but only notable fixes and changes (ignore rt-test fixes, code cleanup updates, parser test updates, etc).
  • Prepare a deploy repo commit and push for +2
    • Roughly: cd deploy ; git checkout master ; git pull origin master ; git submodule update ; cd src ; git checkout {to} ; cd .. ; git add -u ; git commit -m "Bump src to {to} for deploy" ; git review

Verify deployment version on beta after the deploy patch is merged

  • If beta cluster is down or visual editor is down in beta cluster, do not continue with routine deployments.
  • On beta cluster, perform manual VisualEditor editing tests. This requires you to have an account on the beta cluster wiki. Test with non-ASCII content too to catch encoding issues. Check parsoid logs, if necessary.

Be around on IRC

  • Add yourself to the "deployer" field of Deployments if you're not already there
  • Be online in freenode #wikimedia-operations (and stay online through the deployment window)

Deploying the latest version of Parsoid

Before you begin, note that Parsoid caches its git version string. So you may wish to do:

ssh -A tin
tin$ for wtp in `cat /etc/dsh/group/parsoid`; do echo -n "Querying $wtp: "; \
   curl "http://$wtp:8000/_version"; echo; done;

to ensure that the "old" version string is cached, so that you will be able to tell when parsoid restarts with its "new" version below.

Now to do the deploy:

ssh -A tin
tin$ cd /srv/deployment/parsoid/deploy
tin$ git deploy start
tin$ git pull
tin$ git submodule update --init
tin$ git deploy sync

You will then get status updates. If any minions are not ok, then retry the deploy until all are. Proceed with 'y' when all minions are ok for each step.

Nodes are not automatically restarted. First restart Parsoid on a canary (wtp2001 is used here as an example).

ssh wtp2001.codfw.wmnet
wtp2001:~$ sudo service parsoid restart
parsoid stop/waiting
parsoid start/running, process 25849
wtp2001:~$

Add a log entry

 !log synced code; restarted Parsoid on <node-name> as a canary

Monitor wtp2001 on ganglia for a while to make sure things seem okay. (FIXME: describe how to do so). You can curl http://localhost:8000/_version to verify Parsoid on wtp2001 is running correctly. It should output the expected sha. Keep an eye on Kibana for log entries from the node to make sure nothing untoward is happening. Then to restart the rest of the nodes:

tin$ git deploy service restart

This is broken (https://phabricator.wikimedia.org/T102039), so you might need to do this instead (on your localhost):

for wtp in `ssh <your-user-id>@bast1001.wikimedia.org cat /etc/dsh/group/parsoid` ; do echo $wtp ; ssh <your-user-id>@$wtp sudo service parsoid restart ; done

Unfortunately, Parsoid sometimes does not restart cleanly on all nodes. This shows up as an unresponsive node in Ganglia that turns bright red. Log on to that node, and top will tell you the node worker that is stuck. Have a root kill -9 that process. On the rare occasion, there might be more than one stuck process. So, you will have to kill -9 them all. NOTE: This problem is likely going to be fixed soon once https://gerrit.wikimedia.org/r/#/c/268848/ is deployed

Once everything is done, log the deploy in #wikimedia-operations with something like

!log updated Parsoid to version <new hash>

This creates a timestamped entry in the Server Admin Log.

Post-deploy checks

  • Test VE editing on enwiki and non-latin wikis
    • For example, open it:Luna (or other complex page), start the visual editor, make some random vandalism, click save -> review changes, then verify that the wikitext reflects your changes and was not corrupted. Hit cancel to abort the edit.
    • Reading through the recent edits (frwiki, enwiki) can also be a good check.
  • Verify all Parsoid servers are running the same version with:
tin$ for wtp in `cat /etc/dsh/group/parsoid`; do echo -n "Querying $wtp: "; \
   curl "http://$wtp:8000/_version"; echo; done;

Deploying a cherry-picked patch

One way to do this is to create a new branch in the Parsoid repo and cherry-pick your patches to that. For example:

git checkout 497da30e # this is the commit on the master branch that you want to cherry pick on top of
git checkout -b deploy-20150528 # give it a name (go ahead and use the date of your deploy)
git cherry-pick f274c3f54f385a6ac159a47209d279b9040a161c # patch number 1
git cherry-pick de087b106be48fc6e97f2ebc4644f9d297ecdfed # patch number 2
git push gerrit deploy-20150528:deploy-20150528 # create the branch in gerrit (DON'T USE SLASHES HERE)

Now do the usual steps to prepare a deploy repo (see below) using the hash of your branch commit (73445bfd in the example below):

cd deploy
git checkout master ; git pull origin master ; git submodule update ; cd src ; git checkout 73445bfddded9f0baa6afe548c98880f4401fb7b # your branch commit
cd .. ; git add -u ; git commit -m "Bump src to 73445bfd (deploy-20150528 branch) for deploy"
git review -u

Note that the automated push to beta will fail if your gerrit branch name contains a slash. This is probably just because some ancient version of git is being used, and will eventually be fixed. But in the meantime, use dashes instead of slashes.

Cherry-picking directly from tin and deploying it

In many situations, a hotfix might need to be pushed quickly. One way to do that is to cherry-pick the patch on tin and sync it.

### Verify that you have the most recently deployed code that you want to cherry-pick on top of
tin$ cd /srv/deployment/parsoid/deploy (verify via git log)
tin$ cd src (verify via git log)

### Create a hotfix branch
tin$ git checkout -b hotfix_<some_unique_tag>

### Get latest code from master you want to cherry-pick from
tin$ git checkout master; git pull

### Check out the hotfix branch and cherry-pick
tin$ git checkout hotfix_<some_unique_tag>
tin$ git cherry-pick <commit-from-master>

### Create a deploy-repo patch
tin$ cd ..; git commit -a -m "Bump src to whatever-git-sha-it-is for hotfix"

### The usual deployment steps
tin$ git deploy start
tin$ git deploy sync
... restart and verify deployment ...

When something goes wrong

Roan and Gabriel know most about the Parsoid infrastructure. Send them a mail or (if urgent) call if there are issues you can't solve.

Reverting a Parsoid deployment

Code

ssh tin
cd /srv/deployment/parsoid/deploy
git deploy start
git checkout <tag> (all deployed versions are tagged as parsoid/deploy-sync-<date>-<some-id>)
git submodule update --init
git deploy sync

If you want to revert to a specific changeset (should never ever be necessary)

git deploy start
git reset --hard <desired changeset>
git deploy --force sync

You still need to restart the Parsoid service after deploying reverted code. Follow the dsh restart directions below.

Misc stuff

  • Restart parsoid hosts via salt from /srv/deployment/parsoid/deploy: service-restart
  • To abort a deployment after running git deploy start but before git deploy sync , run git deploy abort .
  • There is a lock file preventing multiple deployments on the same code base from being active at the same time. If git deploy start complains about this lock, you can run git deploy abort to make it go away (if you know this isn't a legitimate warning due to someone else actively deploying).
  • If the sync step complains you didn't change anything, you can run git deploy --force sync (note order of arguments!) to make it sync anyway.
  • To change which hosts are pooled or change their weights, edit /home/wikipedia/common/docroot/noc/pybal/eqiad/parsoid as root on fenari

Data flow

Parsoid runs entirely on an internal subnet, so requests to it are proxied through the ve-parsoid API module. This module is implemented in extensions/VisualEditor/ApiVisualEditor.php and is invoked with a POST request to /w/api.php?action=ve-parsoid. The API module then sends a request to Parsoid, either GET /$prefix/$pagename to get the HTML for a page, or POST /$prefix/$pagename to submit HTML and get wikitext back. Parsoid itself also issues requests to /w/api.php to get the wikitext of the requested page and to do template expansion.

Once the ve-parsoid API module receives a response from Parsoid, it either relays it back to the client (when requesting HTML), or saves the returned wikitext to the page (when submitting HTML).

                (POST /w/api.php?action=ve-parsoid)          (GET /en/Barack_Obama?oldid=1234)           (requests for page content and template expansions)
Client browser ------------------------------------------> API ---------------------------->  Parsoid -----------------------------------------------------> API
    ^                                                      | ^                                 |   ^                                                          |
    |                  (response)                          | |      (HTML)                     |   |                   (responses)                            |
    +------------------------------------------------------+ +---------------------------------+   +----------------------------------------------------------+


                (POST /w/api.php?action=ve-parsoid)          (POST /en/Barack_Obama; oldid=1234)
Client browser ------------------------------------------> API ---------------------------->  Parsoid
                                                           | ^                                 |
                                               (save page) | |      (wikitext)                 |
                                                           | +---------------------------------+
                                                           |
                                                        Database