You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Difference between revisions of "Etherpad.wikimedia.org"

From Wikitech-static
Jump to navigation Jump to search
imported>Dzahn
imported>Alexandros Kosiaris
Line 75: Line 75:
On that server run '''sudo apt-cache policy etherpad-lite''' or '''dpkg -l | grep etherpad''' to identify the currently installed version.
On that server run '''sudo apt-cache policy etherpad-lite''' or '''dpkg -l | grep etherpad''' to identify the currently installed version.


Now check which server is the current "build" server which is used to build Debian packages. Check in site.pp for the node with the '''role(builder)''' and ssh to it. (as of 2020-09 this is [[deneb]].codfw.wmnet).
Normally, debian packages are build on the "build" servers and require no access to the internet (which is why our build servers sport internal IPs). In fact our tool of use (pbuilder), even goes into lengths to ensure that the build environment doesn't have internet access. See https://github.com/wikimedia/puppet/tree/production/modules/package_builder#networking for an explanation of the mechanism behind this.


On the build host, run '''apt-get source etherpad-lite''' to get the current package source and cd into the new directory called etherpad-lite-<version>.
==== Build node ====
Of course, with etherpad fetching npm modules during the build time, the above won't work. Instead of using the standard build host, we 've been using a WMCS machine for this. so, ssh into packager01.packaging.eqiad.wmflabs.  


On the build host, run '''git clone https://github.com/ether/etherpad-lite''' to download the current upstream source. You may have to '''export https_proxy="http://webproxy:8080"''' for that to work unless you already set proxies in some dot files in your home dir. cd into the new '''etherpad-lite''' dir and run '''git checkout <VERSION>''' to check out the release branch for your version. Replace <VERSION> with the actual version, f.e. "1.8.5". You can find the current upstream version at https://github.com/ether/etherpad-lite/releases.
==== Step #1, fetch stuff ====
The next step is to clone [https://gerrit.wikimedia.org/r/operations/debs/etherpad-lite].


Copy the contents of the upstream dir over the files in the package source dir: If "1.8.4" is our old version that is '''rsync -av ./etherpad-lite/ ./etherpad-lite-1.8.4/''' where the first dir came from the git clone of upstream and the second dir came from our apt-get source command.  
  git clone
  curl -O https://github.com/ether/etherpad-lite/archive/1.8.6.tar.gz
  cd etherpad-lite
  gbp import-orig ../1.8.6.tar.gz
 
==== Step #2, refresh patches ====
 
First, push all quilt patches
 
  QUILT_PATCHES=debian/patches quilt push -a
 
If you are lucky, they will all apply cleanly. Proceed to step #3 then. Otherwise, you 'll need to mess a bit with those
 
If they did not apply cleanly, you 'll need to refresh them. The push command above will have probably push some but stopped right before pushing the problematic patch. At that point what you need is:
 
* To force push the patch first to get as many hunks applied as possible
 
  QUILT_PATCHES=debian/patches quilt push -f
 
* To then figure out which hunks of the patch did not get applied (quilt will tell you). Then apply manually with an editor and save
 
* Then refresh the patch
 
  QUILT_PATCHES=debian/patches quilt refresh
 
* Repeat the above until all patches have been applied.
 
==== Step #4, bump debian changelog ====


Next, cd into the versioned dir from above and run '''dch -i''' to edit the debian/changelog. Your name and email address should have been added automatically. Edit the version string in the first line. For example if this is the first package for version 1.8.5 then set it to "(1.8.5-1)". Replace UNRELEASED with the actual release name, for example 'buster'. Edit the commit message below with something like "Bump to version 1.8.5". Write/quit the spawned editor to save changes.
Next, cd into the versioned dir from above and run '''dch -i''' to edit the debian/changelog. Your name and email address should have been added automatically. Edit the version string in the first line. For example if this is the first package for version 1.8.5 then set it to "(1.8.5-1)". Replace UNRELEASED with the actual release name, for example 'buster'. Edit the commit message below with something like "Bump to version 1.8.5". Write/quit the spawned editor to save changes.


You will now get a warning that your current directory has been renamed. cd one level up.
==== Step #5, build ====
 
Download the upstream source tarball with '''wget https://github.com/ether/etherpad-lite/archive/1.8.5.tar.gz''' Replace the version with your actual version. (You might have to add https_proxy="http://webproxy.codfw.wmnet:8080"  in front of the command to make it work.)


Rename the file like this: '''mv 1.8.5.tar.gz etherpad-lite_1.8.5.orig.tar.gz''' and switch back into the versioned directory from before.
First, make sure you 've popped all quilt patches


Run '''DIST=buster pdebuild'''.
  QUILT_PATCHES=debian/patches quilt pop -a


Get: "Unmet build dependencies: nodejs (>= 10) npm (>= 5.8) libpq-dev".
Then issue the gbp buildpackage command


FIXME: Resolve the build dependencies (apt-get source nodejs npm libpq-dev ? apt-get build-dep etherpad-lite?)
  GIT_PBUILDER_AUTOCONF=no DIST=buster gbp buildpackage --git-builder=git-pbuilder -us -uc -sa


Now that the package built succesfully, rsync the resulting files over to the current APT repo server. Identify the correct server by looking for '''aptrepo_server''' in '''./hieradata/common.yaml''' in the operations/puppet repo. ssh to it and then pull the files from the build host to the APT repo host. example: '''sudo rsync -va deneb.codfw.wmnet::pbuilder-result/buster-amd64/etherpad* .'''
Assuming everything when according to plan, you should have a package now. tar all the files (source, deb, .dsc, .changes in a simple .tar.gz file). Now that the package built succesfully, copy (via scp -3 perhaps) the resulting files over to the current APT repo server. Identify the correct server by looking for '''aptrepo_server''' in '''./hieradata/common.yaml''' in the operations/puppet repo. example: '''scp -3 packager01.packaging.eqiad.wmflabs:etherpad-lite.deb deneb.codfw.wmnet::pbuilder-result/buster-amd64/etherpad.tar.gz apt1002.wikimedia.org'''


On the APT repo host, use [[reprepro]] to import the package by pointing at the file ending in .changes. example: '''reprepro -C main include buster-wikimedia etherpad-lint...changes'''. See the [[reprepro]] page for more details on how to setup a basedir and GNUPG home to make that work.
On the APT repo host, use [[reprepro]] to import the package by pointing at the file ending in .changes. example: '''reprepro -C main include buster-wikimedia etherpad-lint...changes'''. See the [[reprepro]] page for more details on how to setup a basedir and GNUPG home to make that work.

Revision as of 16:18, 28 September 2020

https://etherpad.wikimedia.org

Note: etherpads are 100% public and open. Anyone can read them. "Obscure names" are never as obscure as you think and are NOT security. Also the etherpad database is not suitable for any long-term storage — don't expect important data to stay there.

Hardware

Running on etherpad1002, a VM on ganeti01.svc.eqiad.wmnet cluster

About

We built our own package dependent on our own nodejs packages. Everything is puppetized.

The database that it uses is on ... just look this up in the puppet site manifest. Cluster m1 as of this writing.

The app runs on port 9000 and requests are reverse proxied by envoy which also terminates SSL.

The EtherpadLite extension (not currently used) allows embedding it into wiki pages.

Database layout

Etherpad-lite has decided to implement a key/value store on top of a RDBMS for some reason. Well it is an abstraction layer so they can work with other backends as well but it seems like the recommended option is an RDDBS (MySQL).

http://etherpad.org/doc/v1.2.1/#index_database_structure seems to be the official documentation (version dependent obviously)

Deleting pads via site admin

To request a deletion, file a security task on Phabricator.

A variety of ways exist (some are not available/do not work):

  1. Deletion through admin and a plugin(we do not have admin and users on purpose for now) so this is ruled out
  2. Deletion through the API https://github.com/ether/etherpad-lite/wiki/HTTP-API (suggested method):
  1. Login to the etherpad host, at the moment, etherpad1001.eqiad.wmnet
  2. Search the API key created on etherpad first start, found on /var/lib/etherpad-lite/APIKEY.txt
  3. Call the deletion api:
curl 'localhost:9001/api/1/deletePad?apikey=<api key gotten from the previous step>&padID=<pad name as used on the URI>'
  1. If everthings is ok, it should respond with {"code":0,"message":"ok","data":null}
  1. Deletion through the CLI https://github.com/ether/etherpad-lite/wiki/Getting-to-know-the-tools-in-bin. Supposedly this should work but it doesn't
  2. Deletion through the DB (this seems to be the only alternative viable option to the API)

Suppose DELETEME is the pad id of the pad you want to remove (pad id can be taken from the url)

 delete from store where `key` like '%DELETEME%'; 

Note that I had good luck deleting pad content via the below, which tosses revisions, chats, and I don't know exactly what the pad2readonly bit is. This is a lot faster than the %DELETEME% query above, now that the db is so bloated.

 delete from store where `key` like 'pad:DELETEME%';
 delete from store where `key` like 'pad2readonly:DELETEME%';

How to list all pads

Two different plugins existed at the time of investigation, one was not installing correctly, one was not of any decent quality

Mediawiki extension

Yes, don't we want to use that and embed in a wiki?

Extension:EtherpadLite

Converting etherpad content into wikitext

  • Small Python script to convert Etherpads into wiki pages - please help turn this into a Toolforge tool!

Maintenance work

Building new debs whenever there are new releases/security patches is the main one here. However since this uses MariaDB misc, also have a look at MariaDB/misc

Upgrading Etherpad version

Etherpad is installed as a Debian package called "etherpad-lite". The puppet role for Etherpad simply installs this package from our own APT repository.

First, identify which server is the current Etherpad server by looking at manifests/site.pp in the operations/puppet git repository. (as of 2020-09 this is etherpad1002.eqiad.wmnet).

On that server run sudo apt-cache policy etherpad-lite or dpkg -l | grep etherpad to identify the currently installed version.

Normally, debian packages are build on the "build" servers and require no access to the internet (which is why our build servers sport internal IPs). In fact our tool of use (pbuilder), even goes into lengths to ensure that the build environment doesn't have internet access. See https://github.com/wikimedia/puppet/tree/production/modules/package_builder#networking for an explanation of the mechanism behind this.

Build node

Of course, with etherpad fetching npm modules during the build time, the above won't work. Instead of using the standard build host, we 've been using a WMCS machine for this. so, ssh into packager01.packaging.eqiad.wmflabs.

Step #1, fetch stuff

The next step is to clone [1].

  git clone 
  curl -O https://github.com/ether/etherpad-lite/archive/1.8.6.tar.gz
  cd etherpad-lite
  gbp import-orig ../1.8.6.tar.gz
  

Step #2, refresh patches

First, push all quilt patches

  QUILT_PATCHES=debian/patches quilt push -a

If you are lucky, they will all apply cleanly. Proceed to step #3 then. Otherwise, you 'll need to mess a bit with those

If they did not apply cleanly, you 'll need to refresh them. The push command above will have probably push some but stopped right before pushing the problematic patch. At that point what you need is:

  • To force push the patch first to get as many hunks applied as possible
 QUILT_PATCHES=debian/patches quilt push -f
  • To then figure out which hunks of the patch did not get applied (quilt will tell you). Then apply manually with an editor and save
  • Then refresh the patch
  QUILT_PATCHES=debian/patches quilt refresh
  • Repeat the above until all patches have been applied.

Step #4, bump debian changelog

Next, cd into the versioned dir from above and run dch -i to edit the debian/changelog. Your name and email address should have been added automatically. Edit the version string in the first line. For example if this is the first package for version 1.8.5 then set it to "(1.8.5-1)". Replace UNRELEASED with the actual release name, for example 'buster'. Edit the commit message below with something like "Bump to version 1.8.5". Write/quit the spawned editor to save changes.

Step #5, build

First, make sure you 've popped all quilt patches

  QUILT_PATCHES=debian/patches quilt pop -a

Then issue the gbp buildpackage command

  GIT_PBUILDER_AUTOCONF=no DIST=buster gbp buildpackage --git-builder=git-pbuilder -us -uc -sa

Assuming everything when according to plan, you should have a package now. tar all the files (source, deb, .dsc, .changes in a simple .tar.gz file). Now that the package built succesfully, copy (via scp -3 perhaps) the resulting files over to the current APT repo server. Identify the correct server by looking for aptrepo_server in ./hieradata/common.yaml in the operations/puppet repo. example: scp -3 packager01.packaging.eqiad.wmflabs:etherpad-lite.deb deneb.codfw.wmnet::pbuilder-result/buster-amd64/etherpad.tar.gz apt1002.wikimedia.org

On the APT repo host, use reprepro to import the package by pointing at the file ending in .changes. example: reprepro -C main include buster-wikimedia etherpad-lint...changes. See the reprepro page for more details on how to setup a basedir and GNUPG home to make that work.

Run sudo -E reprepro ls etherpad-lite to confirm the new version has been imported.

Switch to the etherpad host itself and run sudo apt-get update and sudo apt-get install etherpad-lite (optionally you can first add -n to simulate an install without actually doing it).

Confirm things are still working. Done.

See also