You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Parsoid: Difference between revisions
imported>Subramanya Sastry |
imported>C. Scott Ananian (→Verify deployment version on beta after the vendor patch is merged: Add -C0 to the grep command to reduce noise.) |
||
Line 51: | Line 51: | ||
* Tag a new version of parsoid and push the tag: | * Tag a new version of parsoid and push the tag: | ||
** <code>git tag v0.12.0-a{N}</code> then <code>git push origin v0.12.0-a{N}</code> (Include the leading 'v', and substitute the next version number for ''{N}''.) | ** <code>git tag v0.12.0-a{N}</code> then <code>git push origin v0.12.0-a{N}</code> (Include the leading 'v', and substitute the next version number for ''{N}''.) | ||
*** The "origin" remote here is <code><nowiki>ssh://USER@gerrit.wikimedia.org:29418/mediawiki/services/parsoid</nowiki></code> | |||
*** Nothing more than usual <code>push</code> permissions on the parsoid repo should be needed. If you need to tweak permissions (for example, to temporarily add <code>force push</code> permissions to fix a mistake), you can do this using the "edit" button at https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/services/parsoid,access | |||
** Check that this version has been picked up at https://packagist.org/packages/wikimedia/parsoid (might take a minute) | ** Check that this version has been picked up at https://packagist.org/packages/wikimedia/parsoid (might take a minute) | ||
* Checkout <code>mediawiki/vendor.git</code> '''master''' branch. In that repo: | * Checkout <code>mediawiki/vendor.git</code> '''master''' branch. In that repo: | ||
** Update <code>composer.json</code> to include <code>"wikimedia/parsoid": "0.12.0-aN",</code> (for your version ''{N}''; note no leading "v") | ** Update <code>composer.json</code> to include <code>"wikimedia/parsoid": "0.12.0-aN",</code> (for your version ''{N}''; note no leading "v") | ||
** Ensure you're running the version of composer listed [https://github.com/wikimedia/mediawiki-vendor#adding-or-updating-libraries in the README for the vendor repo]. At time of writing this is `1.10.0`. `composer --version` will tell you what version you're running and (usually) `composer self-update` will bring you up-to-date. | |||
** Do <code>composer update --no-dev</code> (which should only update parsoid) | ** Do <code>composer update --no-dev</code> (which should only update parsoid) | ||
*** If composer complains "<code>The requested package wikimedia/parsoid 0.12.0-aN exists as [...long list not including 0.12.0-aN...]</code>" then composer's local cache hasn't been updated to include the new version available from packagist yet. Wait a minute and try again. The <code>--no-cache</code> option to composer *might* help... but it might not (it might be an upstream cache). A few minutes always helps. | |||
** Add the changed files to git, commit, and upload to gerrit: | ** Add the changed files to git, commit, and upload to gerrit: | ||
<syntaxhighlight> | <syntaxhighlight> | ||
Line 70: | Line 74: | ||
Bug: T222222 | Bug: T222222 | ||
</pre> | </pre> | ||
* If jenkins fails on gerrit with the same "<code>The requested package wikimedia/parsoid 0.12.0-aN exists as...</code>" message described above, the reason is the same: composer's cache on jenkins still doesn't have your new version yet. Wait a minute and comment "recheck" to re-run the jenkins tests. | |||
* Review and C+2 on gerrit. This will go live on beta cluster pretty quickly (within 30 minutes). | * Review and C+2 on gerrit. This will go live on beta cluster pretty quickly (within 30 minutes). | ||
* '''TEMPORARILY''' you will also need to ssh to <code>deployment-parsoid11.deployment-prep.eqiad.wmflabs</code> and do a scap pull due to [[phab:T247545]] | <!-- This shouldn't be necessary any more... | ||
* '''TEMPORARILY''' you will also need to ssh to <code>deployment-parsoid11.deployment-prep.eqiad.wmflabs</code> and do a scap pull due to [[phab:T247545]] --> | |||
==== Verify deployment version on beta after the vendor patch is merged ==== | ==== Verify deployment version on beta after the vendor patch is merged ==== | ||
* Check that this is live on the mediawiki front ends in beta by watching the version number listed on https://en.wikipedia.beta.wmflabs.org/wiki/Special:Version#mw-version-library-wikimedia | * Check that this is live on the mediawiki front ends in beta by watching the version number listed on https://en.wikipedia.beta.wmflabs.org/wiki/Special:Version#mw-version-library-wikimedia/parsoid | ||
* '''TEMPORARILY''' while we're having scap problems, you'll want to check the version on <code>parsoid11</code> as well: | * '''TEMPORARILY''' while we're having scap problems, you'll want to check the version on <code>parsoid11</code> as well: | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
$ ssh deployment-parsoid11.deployment-prep.eqiad.wmflabs | $ ssh deployment-parsoid11.deployment-prep.eqiad.wmflabs | ||
user@deployment-parsoid11$ curl -x deployment-parsoid11:80 'http://en.wikipedia.beta.wmflabs.org/wiki/Special:Version' | fgrep wikimedia/parsoid | user@deployment-parsoid11$ curl -x deployment-parsoid11:80 'http://en.wikipedia.beta.wmflabs.org/wiki/Special:Version' | fgrep wikimedia/parsoid -C0 | ||
</syntaxhighlight> | </syntaxhighlight> | ||
* If beta cluster is down or visual editor is down in beta cluster, do not continue with routine deployments. | * If beta cluster is down or visual editor is down in beta cluster, do not continue with routine deployments. |
Revision as of 19:09, 20 March 2020
Parsoid is a service that parses converts between wikitext and HTML. The HTML contains additional metadata that allows it to be converted back ("round-tripped") to wikitext.
- VisualEditor fetches the HTML for a given page from Parsoid, edits it, then delivers the modified HTML to Parsoid, which converts it back to wikitext. Parsoid is a stateless HTTP server running on port 8000.
- Flow (as configured on WMF wikis with
$wgFlowContentFormat = 'html'
) works the other way around. When a user creates a post Flow uses Parsoid to convert the wikitext to HTML and Flow stores the HTML in ExternalStore. If someone later edits a post Flow uses Parsoid to convert the HTML back to wikitext for editing.
Monitoring
- Parsoid eqiad cluster in Grafana
- Parsoid codfw cluster in Grafana
- Icinga has service checks for HTTP on port 8000 on both the individual backends and on the LVS service IP.
- pybal does health checks on all backends every second, and depools boxes that are down as long as the % of depooled boxes does not exceed 50%. To see these health checks and depools/repools happen in real time, run
ssh parsoid.svc.eqiad.wmnet
(this will drop you into either lvs1003 or lvs1006, depending on which is active), thentail -f /var/log/pybal.log | grep parsoid
- Logging happens in /var/log/parsoid/parsoid.log. There is a log rotation setup in /etc/logrotate.d/parsoid.
- Logs in logstash (Parsoid/PHP): https://logstash.wikimedia.org/app/kibana#/dashboard/AW4Y6bumP44edBvO7lRc
- Useful links for Parsoid deployers
- Currently running Parsoid version:
- In beta: https://en.wikipedia.beta.wmflabs.org/wiki/Special:Version#mw-version-library-wikimedia.2Fparsoid
- In production: https://en.wikipedia.org/wiki/Special:Version#mw-version-library-wikimedia.2Fparsoid
- On scandium: See https://www.mediawiki.org/wiki/Parsoid/Round-trip_testing
- (Be aware that the parsoid cluster is behind restbase and although the cluster *should* be running the same version of Parsoid as the mediawiki frontends, if puppet or scap are broken (esp in beta) things could diverge.)
Deploying changes
Parsoid is deployed as part of the MediaWiki train. See How to deploy code for an overview, Heterogeneous deployment for a more technical description of the directory structures involved, and Heterogeneous deployment/Train deploys for the steps to do a train deploy. When code changes outside the train schedule are required, a SWAT deploy will be required. Generally Parsing team members won't be doing train deploys or SWAT deploys directly; we will tag a Parsoid version (which releases it to packagist to make it available via composer) and merge a version bump into the mediawiki/vendor repository. Once the patch is merged into vendor
, the new version of Parsoid goes live in beta (almost) immediately; it will then be rolled out to production on the next train.
Machine overview
These are the machines involved in a Parsoid deploy:
- In the beta/wmflabs cluster:
deployment-deploy01.deployment-prep.eqiad.wmflabs
: staging host in beta; no longer used.deployment-parsoid11.deployment-prep.eqiad.wmflabs
: parsoid server in betadeployment-restbase02.deployment-prep.eqiad.wmflabs
: restbase server in beta
- In the production cluster:
deployment.eqiad.wmnet
: staging host in production; no longer usedwtp1xxx
: parsoid servers in eqiad clusterrestbase1xxx
: restbase servers in eqiad clusterwtp2xxx
: parsoid servers in codfw clusterrestbase2xxx
: restbase servers in codfw clusterscandium.eqiad.wmnet
: Parsoid testing host, has read-only access to the production database.
Deploying Parsoid
Test the version you hope to deploy
- See mw:Parsoid/Round-trip testing for details.
- RT testing results are no longer publicly accessible on the web. You need to establish a ssh tunnel to the web service on scandium.
- Check http://localhost:<tunnel-port>/regressions/between/{from}/{to} where {from} is the last deployed hash from mw:Parsoid/Deployments and {to} is the latest tested commit (which we're about to deploy)
- http://localhost:<tunnel-port>/commits gives you a nice radio-button interface to create this URL
- BEWARE: if you get the output
total regressions between selected revisions: 0
, it is extremely likely that you mistyped the hash or that we didn't actually run round-trip tests for that particular hash. (This is a bug, we should probably give a better message in this case.) - Since we are using current revision of titles in round-trip testing, edits to pages can show up as false regressions. tools/regression-testing.js is useful in filtering those out. Grab the wiki:title pairs that show up as regressions in the regressions url, save it to a file and feed it to the regressions tool and you will get a list of pages to look more closely, if necessary.
- Check that there are no concerning notices or errors in logstash from the rt run
Prepare the vendor patch
(This process was hashed out in phab:T240055)
- Create a short deployment summary on mw:Parsoid/Deployments from
git log --cherry-pick {from}...{to}
. Don't include all commits, but only notable fixes and changes (ignore rt-test fixes, code cleanup updates, parser test updates, etc). (The above command will do the right thing if {from} was on a branch and had patches cherry-picked from {to}, although if there were conflicts during the cherry-pick to {from} the patch will still appear in the log for {to}.) - Tag a new version of parsoid and push the tag:
git tag v0.12.0-a{N}
thengit push origin v0.12.0-a{N}
(Include the leading 'v', and substitute the next version number for {N}.)- The "origin" remote here is
ssh://USER@gerrit.wikimedia.org:29418/mediawiki/services/parsoid
- Nothing more than usual
push
permissions on the parsoid repo should be needed. If you need to tweak permissions (for example, to temporarily addforce push
permissions to fix a mistake), you can do this using the "edit" button at https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/services/parsoid,access
- The "origin" remote here is
- Check that this version has been picked up at https://packagist.org/packages/wikimedia/parsoid (might take a minute)
- Checkout
mediawiki/vendor.git
master branch. In that repo:- Update
composer.json
to include"wikimedia/parsoid": "0.12.0-aN",
(for your version {N}; note no leading "v") - Ensure you're running the version of composer listed in the README for the vendor repo. At time of writing this is `1.10.0`. `composer --version` will tell you what version you're running and (usually) `composer self-update` will bring you up-to-date.
- Do
composer update --no-dev
(which should only update parsoid)- If composer complains "
The requested package wikimedia/parsoid 0.12.0-aN exists as [...long list not including 0.12.0-aN...]
" then composer's local cache hasn't been updated to include the new version available from packagist yet. Wait a minute and try again. The--no-cache
option to composer *might* help... but it might not (it might be an upstream cache). A few minutes always helps.
- If composer complains "
- Add the changed files to git, commit, and upload to gerrit:
- Update
git add wikimedia/parsoid composer.lock composer.json # & etc
git commit
git review
- Use a commit message that (1) names the new parsoid tag, (2) includes the git hash of the new parsoid version, and (3) references key bug #s from the deployment summary so the deploy gets linked to phab. For example:
Bump parsoid to 0.12.0-aN This corresponds to Parsoid commit cafecafecafecafe. Bug: T111111 Bug: T222222
- If jenkins fails on gerrit with the same "
The requested package wikimedia/parsoid 0.12.0-aN exists as...
" message described above, the reason is the same: composer's cache on jenkins still doesn't have your new version yet. Wait a minute and comment "recheck" to re-run the jenkins tests. - Review and C+2 on gerrit. This will go live on beta cluster pretty quickly (within 30 minutes).
Verify deployment version on beta after the vendor patch is merged
- Check that this is live on the mediawiki front ends in beta by watching the version number listed on https://en.wikipedia.beta.wmflabs.org/wiki/Special:Version#mw-version-library-wikimedia/parsoid
- TEMPORARILY while we're having scap problems, you'll want to check the version on
parsoid11
as well:
$ ssh deployment-parsoid11.deployment-prep.eqiad.wmflabs
user@deployment-parsoid11$ curl -x deployment-parsoid11:80 'http://en.wikipedia.beta.wmflabs.org/wiki/Special:Version' | fgrep wikimedia/parsoid -C0
- If beta cluster is down or visual editor is down in beta cluster, do not continue with routine deployments.
- On beta cluster (eg
en.wikipedia.beta.wmflabs.org
), perform manual VisualEditor editing tests. This requires you to have an account on the beta cluster wiki. Test with non-ASCII content too to catch encoding issues. Check parsoid logs. Be particularly alert to integration issues: library conflicts, etc. - Watch the logs on beta: https://wikitech.wikimedia.org/wiki/Logstash#Beta_Cluster_Logstash
Be around on IRC
This hasn't been updated for the train yet. But in general we need parsing team coverage during the period when the train is rolling forward, in order to catch any Parsoid-related issues.
- Add yourself to the "deployer" field of Deployments if you're not already there
- Be online in freenode #wikimedia-operations (and stay online through the deployment window)
During the Parsoid deploy
This hasn't been fully updated for train process yet. In general we're not doing the scap and we no longer have canary nodes to watch. But we can/should be monitoring logs as the ops team rolls the train forward.
Group 0
- Check the parsoid version on mediawiki.org to ensure parsoid has been deployed to group 0.
- Worth verifying the parsoid version in the cluster by ssh to a canary:
$ ssh wtp1025.eqiad.wmnet
user@wtp1025:~$ curl -x wtp1025:80 'http://www.mediawiki.org/wiki/Special:Version' | fgrep wikimedia/parsoid -C0
- TBD: test edits
- TBD: log links
Group 1
- TBD
Logs to monitor
Post-deploy checks
- Test VE editing on enwiki and non-latin wikis
- For example, open it:Luna (or other complex page), start the visual editor, make some random vandalism, click save -> review changes, then verify that the wikitext reflects your changes and was not corrupted. Hit cancel to abort the edit.
- Reading through the recent edits (frwiki, enwiki) can also be a good check.
Testing a version bump
If the deployed version of Parsoid updates the Parsoid DOM version and/or will exercises the html2html "down convert" endpoint, the following test procedure will ensure that clients are getting the appropriate DOM version:
- First and foremost, mocha tests should already be present that cover both downgrading the HTML and serializing it with and without selser.
- Create a test page on the beta cluster containing the features that merited the major version bump.
- Deploy the desired commit to the beta cluster and, as a sanity check, make requests for the above test page from Parsoid directly (via
deployment-parsoid11.deployment-prep.eqiad.wmflabs
) accepting the various specs that are available. The inline meta tag and aforementioned features should indicate that it worked. Example requests might be,- For the old version,
curl -x deployment-parsoid11:80 'http://en.wikipedia.beta.wmflabs.org/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/Test_Page' -H'Accept: text/html; charset=utf-8; profile="https://www.mediawiki.org/wiki/Specs/HTML/1.7.0"'
- For the new version,
curl -x deployment-parsoid11:80 'http://en.wikipedia.beta.wmflabs.org/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/Test_Page' -H'Accept: text/html; charset=utf-8; profile="https://www.mediawiki.org/wiki/Specs/HTML/2.0.0"'
- For the old version,
- Confirm that VE on the beta cluster is still tied to the older content version and will be needing a downgrade (see the commit in Special:Version for the extension and compare with the header defined in
includes/ApiVisualEditor.php
) - At this point, two scenarios need to be tested: an edit starting from the older content version stored in RESTBase (which won't require a downgrade) and one starting from the new content version, which will.
- Note that, for extra points, there are potentially several versions numbers stored in RESTBase that satisfy the VE request based on caret semantics and it might be worthwhile to confirm that edits starting from those versions work as well.
- Once you've found stored content in RESTBase with an appropriate version for your test it's prudent to confirm that VE is actually editing what you expect. This can be achieved by dumping the various DOMs: the original
copy(ve.init.target.doc.body.outerHTML)
and the editedcopy(ve.init.target.docToSave.body.outerHTML)
- In each case, try to confirm that the features can be edited directly as well as being ignored by selser (usually because no normalizations occur). Unfortunately, testing here is a bit more art than science.
- Finally, open up the various testing dashboards for logging and metrics to verify that no unexpected errors are present and that the downgrades are accounted for.
Testing on scandium
When on scandium, use this command to test Parsoid directly:
curl -x scandium.eqiad.wmnet:80 http://<domain>/w/rest.php/<domain>/v3/page/html/<title>/<revid>
Testing LanguageConverter
LanguageConverter can be tested on beta in a manner similar to testing a version bump.
- Create a test page on the beta cluster containing the language converter features you wish to touch. Either the page language for the article must be set to a language w/ variants, or else the article must take place on a wiki where the main language has variants. We'll use the SrTest page on beta srwiki in our examples below.
- Deploy the desired commit to the beta cluster and, as a sanity check, make requests for the above test page from Parsoid directly (via ssh to
deployment-parsoid11.deployment-prep.eqiad.wmflabs
) specifying the desired variant language. Verify that the result has been converted appropriately. Example requests might be,
curl -H'Accept-Language: sr-ec' -x deployment-parsoid11:80 http://sr.wikipedia.beta.wmflabs.org/w/rest.php/sr.wikipedia.beta.wmflabs.org/v3/page/html/User:Cscott%2FSrTest/23
curl -H'Accept-Language: sr-el' -x deployment-parsoid11:80 http://sr.wikipedia.beta.wmflabs.org/w/rest.php/sr.wikipedia.beta.wmflabs.org/v3/page/html/User:Cscott%2FSrTest/23
- To test in production, try something like:
curl -X GET --header 'Accept-Language: sr-el' 'https://sr.wikipedia.org/api/rest_v1/page/html/%D0%93%D0%BB%D0%B0%D0%B2%D0%BD%D0%B0_%D1%81%D1%82%D1%80%D0%B0%D0%BD%D0%B0/21280369'
See https://phabricator.wikimedia.org/T241146#5810424 for some more examples.
Deploying a cherry-picked patch
One way to do this is to create a new branch in the Parsoid repo and cherry-pick your patches to that. For example:
git checkout v0.12.0-a3 # this is the commit on the master branch that you want to cherry pick on top of git checkout -b deploy-20150528 # give it a name (go ahead and use the date of your deploy) git cherry-pick f274c3f54f385a6ac159a47209d279b9040a161c # patch number 1 git cherry-pick de087b106be48fc6e97f2ebc4644f9d297ecdfed # patch number 2 git push gerrit deploy-20150528:deploy-20150528 # create the branch in gerrit (DON'T USE SLASHES HERE)
Now do the usual steps to tag a release and prepare a vendor branch patch (see above) using the next available release version number (v0.12.0-a4 in the example below):
git tag v0.12.0-a4 # this is the next available release number git push origin v0.12.0-a4
Switch to the mediawiki/vendor
repository:
git checkout master ; git pull origin master edit composer.json # set wikimedia/parsoid to v0.12.0-a4 composer update --no-dev git add -u git commit -m "Bump wikimedia/parsoid to v0.12.0-a4" git review -u
Note that the automated push to beta will fail if your gerrit branch name contains a slash. This is probably just because some ancient version of git is being used, and will eventually be fixed. But in the meantime, use dashes instead of slashes.
Misc stuff
- To deploy to a single host
scap deploy --force -l <node>
- To see which hosts are pooled, from another host
confctl select dc=.*,cluster=parsoid,service=parsoid get
- To see the list of parsoid hosts in beta:
cat /srv/deployment/parsoid/deploy/scap/betacluster
- See also
/srv/deployment/parsoid/deploy/scap/scap.cfg
in general
- To pool/depool a node, from deployment.eqiad.wmnet, run:
- To depool:
SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l deploy-service <node> 'depool service=parsoid'
- To pool :
SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l deploy-service <node> 'pool service=parsoid'
- To depool:
Data flow
Parsoid runs entirely on an internal subnet, so requests to it are proxied through the ve-parsoid API module. This module is implemented in extensions/VisualEditor/ApiVisualEditor.php
and is invoked with a POST request to /w/api.php?action=ve-parsoid
. The API module then sends a request to Parsoid, either GET /$prefix/$pagename
to get the HTML for a page, or POST /$prefix/$pagename
to submit HTML and get wikitext back. Parsoid itself also issues requests to /w/api.php
to get the wikitext of the requested page and to do template expansion.
Once the ve-parsoid API module receives a response from Parsoid, it either relays it back to the client (when requesting HTML), or saves the returned wikitext to the page (when submitting HTML).
(POST /w/api.php?action=ve-parsoid) (GET /en/Barack_Obama?oldid=1234) (requests for page content and template expansions) Client browser ------------------------------------------> API ----------------------------> Parsoid -----------------------------------------------------> API ^ | ^ | ^ | | (response) | | (HTML) | | (responses) | +------------------------------------------------------+ +---------------------------------+ +----------------------------------------------------------+ (POST /w/api.php?action=ve-parsoid) (POST /en/Barack_Obama; oldid=1234) Client browser ------------------------------------------> API ----------------------------> Parsoid | ^ | (save page) | | (wikitext) | | +---------------------------------+ | Database