Difference between revisions of "Portal:Data Services/Admin/Wiki Replicas"

From Wikitech-static
Jump to navigation Jump to search
(→‎Step 6: setting views: Clarify a bit)
(→‎De-pooling Replica Servers: move depooling to a runbook)
Line 334: Line 334:

=== De-pooling Replica Servers ===
=== De-pooling Replica Servers ===
See [[Portal:Data Services/Admin/Runbooks/Depool wikireplicas]]
Depooling is generally done on the two physical dbproxy servers for both legacy and multi-instance services.
==== Legacy replicas ====
* This is accomplished on the proxy level (currently dbproxy1018 for analytics and dbproxy1019 for web)
* To actually depool each server:
** If you want to depool labsdb1009 you'd need to edit [[phab:source/operations-puppet/browse/production/hieradata/hosts/dbproxy1019.yaml|hieradata/hosts/dbproxy1019.yaml]] and basically switch IPs and hostnames to leave lasbsdb1010 as primary.
** If you want to depool labsdb1010 you'd need to edit [[phab:source/operations-puppet/browse/production/hieradata/hosts/dbproxy1018.yaml|hieradata/hosts/dbproxy1018.yaml]] and delete (or comment) the lines, 3-5.
** If you want to depool labsdb1011 you'd need to edit [[phab:source/operations-puppet/browse/production/hieradata/hosts/dbproxy1018.yaml|hieradata/hosts/dbproxy1018.yaml]] and delete (or comment) the lines, 6-8.
* <u>The actual merge and reload of the proxy should be coordinated with the DBAs.</u> They may have more information than we do about other maintenance that is already in progress which needs to be considered before taking new actions. It is usually enough to check with them via irc to see if there is any blocker to activating the change.
* To repool, simply revert the patch, merge, and reload. Communication with the DBAs at this point is advised as well so that they can track the global state of things more easily.
* You can check what haproxy thinks is up or down as root by running <code>echo "show stat" | socat /run/haproxy/haproxy.sock stdio</code> and sifting through what that dumps out.
==== Multi-instance replicas ====
Since there is much more complexity in the multi-instance setup, the basic, default configuration is pulled from puppetdb based on puppetized settings of the database servers themselves. Each analytics database instance is the standby for the web service of the same section number, and each web database instance is the standby for the analytics instance of the same section number, using some reduce and merge trickery in [[phab:source/operations-puppet-browse/production/modules/profile/manifests/mariadb/proxy/multiinstance_replicas.pp|puppet]]. Depooling is accomplished by overriding what the reduce functions produce for the sections in the resulting hash using the <code>profile::mariadb::proxy::multiinstance_replicas::section_overrides</code> key in host hiera, which will be merged at the end of the process.
An example of doing this is provided in the comments of each proxy server's host hiera file to make it clearer and for quicker depooling. Once the appropriate hiera file is changed, you will need to do the same reloading process as for the legacy replicas for this to take effect.
As a basic example, if you wanted to depool clouddb1015.eqiad.wmnet specifically and it is ok to leave it as the secondary on dbproxy1018 (which it normally would be), then you'd add (or decomment) the following to hieradata/hosts/dbproxy1019.yaml and you'd need to '''reload haproxy''' on the given proxy, and this should do it by simply overwriting the s4 and s6 keys:
<syntaxhighlight lang=yaml>
Besides <code>ipaddress</code>, other valid keys are <code>weight</code>, which only really matters if you have more than one host under the section key, the boolean <code>standby</code>, which adds the host entry to haproxy as a standby under that key and <code>depooled</code>, which expressly removes the host entry from the resulting file. The last one is unlikely to be necessary since adding an uncommented section key (eg. s2:) will overwrite the automatically generated one anyway. It could be useful one day, possibly if we start doing deep merges or something for the overrides.  At this time, it is a shallow merge.
Depooling an entire back proxy (eg. dbproxy1018) is done via the horizon hiera on the VM proxy. Each section is pointed at the appropriate back proxy's LVS ip address. If you were to change that for each section to the LVS IP of dbproxy1019 on both VM proxies, for example, all requests would go to only one of the back proxies (dbproxy1019 in this example).

=== Making significant changes to the views and the indexes ===
=== Making significant changes to the views and the indexes ===

Revision as of 23:37, 30 September 2021

This page holds most of the knowledge we have regarding Wiki Replicas for admins.

Service architecture by layers

Description of how the service is currently deployed.

Legacy system diagram


The diagram shows the different network connections and data flows as they cross different realms and devices.

This diagram no longer represents the reference architecture for the wiki replicas services going forward, and is left here just as a reference.

Multi-instance system diagram

The diagram above shows the 2 proxy layers that live in front of the replicas DB servers.

The proxy layer 1 are virtual machines in CloudVPS, which abstract away port numbers for each database section.

The proxy layer 2 are physical servers in the wiki production realm that know the actual internal address of each replica DB server.

Physical database server layer

Legacy Wikireplicas

This service is build on top of several physical servers:

All servers are the same (except labsdb1012, which is dedicated to the Analytics team):

  • HP ProLiant DL380 Gen9
  • 16 CPU Intel(R) Xeon(R) CPU E5-2637 v3 @ 3.50GHz
  • 500GB RAM
  • 1 hard-disc in /dev/sda with 11.7 TiB (probaby RAID, using hpsa kernel driver)
  • 4 x 1GB ethernet interfaces (tg3)

All monitoring, including RAID status, is done by icinga.

labsdb1012 is a HP ProLiant DL380 Gen10 (2U) and is using Intel(R) Xeon(R) Silver 4112 CPUs instead. This server will become multi-instance soon, but it will still be dedicated to the Analytics team's use.

Multiinstance Wikireplicas

There are currently 8 servers in this cluster. Each of the four hosts two database sections. Since there are 8 sections in the scope of the Wikireplicas service, that means that they are able to function as sets of four that can act as standby for one another. Below are the servers, organized by the service (web or analytics):

Web (realtime workloads)
Analytics (long-processing workloads)

These are all Dell PowerEdge R440s, built to Data Persistence specs.

Physical proxy layer

Two haproxy servers are the front proxy for connections to the legacy service and the back proxy in front of the replica servers for the multi-instance hosts (behind the LVS layer).

DB layer

All database servers run MariaDB.

The legacy databases each have multi-source replication happening so that each server is capable of hosting sections s1 through s8, effectively hosting sanitized replica material for all in-scope wikis in one database instance.

The multi-instance servers have a somewhat more complex layout that is documented in each database server's host hiera file (eg. clouddb1013.yaml). They are multi-instance in the sense that each server has two actual mariadb database instances running at the same time on nonstandard ports (see: profile::mariadb::section_ports). Each instance is named for and hosts one database section.

Higher layers

Such as applications running on top, proxies, caches, et al.


main article Help:Toolforge/Database#Naming conventions

Legacy wikireplicas

The DNS names of the wiki-replicas databases as seen from the SQL client point of view are things like:

  • ${PROJECT}.{analytics,web}.db.svc.eqiad.wmflabs. Example: enwiki.web.db.svc.eqiad.wmflabs
  • s${SHARD_NUMBER}.{analytics,web}.db.svc.eqiad.wmflabs. Example: s1.analytics.db.svc.eqiad.wmflabs
Multi-instance wikireplicas

DNS is an important component of helping our users find their server in multi-instance. Each wiki database has a CNAME in the form of:

  • ${WIKI-DATABASE}.{analytics,web}.db.svc.wikimedia.cloud. Example enwiki.web.db.svc.wikimedia.cloud
  • s${SECTION_NUMBER}.{analytics,web}.db.svc.wikimedia.cloud. Example s1.web.db.svc.wikimedia.cloud

By connecting to the wiki database's name, the CNAME redirects the user to the appropriate section's DNS, which is a distinct IP address that will route to the appropriate port on the physical proxies. This way, instead of users needing to figure out that diqwiktionary is on s5 (by checking here perhaps) and then determining from puppet that s5 uses the port 3315 at the proxy and connecting to that, they just need to know that they are connecting to diqwiktionary and DNS plus two proxies handle the rest. This allows components behind this layer to be depooled and maintained simply without a need to provide lengthy explanations of our infrastructure for very basic usage.

More advanced users who wish to attempt a cross-wiki join could determine the section number of two wiki databases using the dig tool:

bstorm@tools-sgebastion-08:~$ dig +short diqwiktionary.analytics.db.svc.wikimedia.cloud
bstorm@tools-sgebastion-08:~$ dig +short avkwiki.analytics.db.svc.wikimedia.cloud
bstorm@tools-sgebastion-08:~$ dig +short eswiki.analytics.db.svc.wikimedia.cloud

By running the above, they could determine that a cross-wiki SQL join between diqwiktionary and avkwiki is possible because they are both on s5, while they cannot be joined to eswiki because that is on s7. This is much faster, programmatically, than scanning noc.wikimedia.org, depending on the use case. That information can sometimes be gained from meta_p (which is hosted on s7), and always from the dblists files on noc.wikimedia.org if checking the section for what it contains rather than check where a database lives.

VM proxies

For the multi-instance wikireplicas service, the routing of requests first hits a pair of VMs that each act as a proxy for one of the web or analytics services. The current proxies are:

Each is an haproxy server that holds 8 VIPs that are reserved as Neutron ports, connected to the VM via puppet and ends up in DNS via wikireplicas_dns.yaml in puppet. Each IP is the frontend for a section. Each section then routes to the LVS IP address for the upstream hardware proxy servers (dbproxy1018 and dbproxy1019 at this time) on the appropriate port. The hardware proxies handle load balancing in the default configuration. This layer purely routes an IP address for a section at port 3306 to the section's port on the multi-instance cluster and is discoverable via the wiki database name in DNS.

From production to wiki-replicas

Wiki databases from production are copied and sanitized to serve wiki-replicas.

Step 0: databases in production

The starting situation is that there are databases in production for wiki projects (like wikipedia, wikidata, wikictionary, and friends). We would like to provide this same databases for WMCS users. Due to privacy reasons, some data needs to be redacted or deleted. That's why users can't directly access this database.

So, we choose what databases to copy to wiki-replicas, which are all of them.

Every time a new database is created in production (for example, a new language for a wiki) we are in this step 0.

The identification of new database candidates for migrating to wiki-replicas is done under request by someone. Right now there aren't any mechanisms to notify pending migrations or the like.

Step 1: sanitization

Main article: MariaDB/Sanitarium and Labsdbs

The production database is copied to sanitarium boxes by means of MariaDB replication mechanisms (TODO: is this true? give more info if possible).

Each sanitarium host has a MariaDB instance to replicate each db shard. The replication into the sanitarium host uses triggers and filters to remove sensitive columns, tables and databases in the simple case where there are no conditions (e.g. Ensures user_password does not go into Cloud Services).

Having this redaction done on a separate host outside of Cloud Services helps isolate the security of the data and ensure a privilege escalation via the Cloud Services access does not compromise the most sensitive data in the db.

Some triggers are added by means of:

This step is handled by main operations teams, DBAs.

Step 2: evaluation

Main article: Labsdb redaction

Once data is in sanitarium boxes, some cron jobs and manual scripts check whether data is actually redacted. For example, check that a given column is NULL.

Involved code:

This evaluation also happens in the wiki-replica servers, and alerts in case some private data is detected.

The main production team, DBAs, are in charge of this step.

TODO: where are the cron jobs?

Step 3: filling up wiki-replicas

Data is finally copied to wiki-replica servers. The DB servers are currently using row-based replication from sanitization boxes to wiki-replicas.

This is done in real time by the DB daemons, so there is always a data flow, which should be sanitized due to previous steps.

Step 4: setting up GRANTs

At this point, database GRANTS are created by main operations team DBAs.

This is done by hand, no automation using puppet, but there is a file where GRANTS are being tracked: modules/role/templates/mariadb/grants/wiki-replicas.sql.

The content of the file is something like the following:

Related tickets:

Step 5: setting indexes

Create indexes that only exist on the wiki replicas via the maintain-replica-indexes script. This script manages the indexes in an idempotent fashion (if you run it multiple times it only changes what doesn't match the configuration file here index-conf.yaml). This could use the ability to handle only a single database, but it skips tables it finds comply with the definitions in the config, so it is fairly quick to run, despite that it runs across the whole set.

On each wiki-replica server, run:

% sudo /usr/local/sbin/maintain-replica-indexes

Step 6: setting views

Create _p (public) views, which are intermediate views which leave out private data.

This is done by means of the maintain-views.py script.

In each wiki-replica server, it's executed like this:

% sudo maintain-views --databases $wiki --debug

This step is handled by the WMCS team.

TODO: elaborate info on what is this doing

Step 7: setting up metadata

Insert a new row in meta_p.wiki for the new wiki by running the maintain-meta_p script.

The execution is like this:

% sudo /usr/local/sbin/maintain-meta_p --databases $wiki. 

This step is handled by the WMCS team.

TODO: elaborate info on what is this doing

Step 8: setting up DNS

From an openstack control server (for example cloudcontrol1004.wikimedia.org), run:

$ sudo /usr/local/sbin/wikireplica_dns --aliases --shard <sN>

To update the service IP used for the analytics or web servers:

  • Edit modules/openstack/files/util/wikireplica_dns.yaml to change the service IP
  • Run /usr/local/sbin/wikireplica_dns
root@labcontrol1001:~# /usr/local/sbin/wikireplica_dns -v --zone web.db.svc.eqiad.wmflabs.
2018-03-16T14:02:41Z mwopenstackclients.DnsManager INFO    : Updating s8.web.db.svc.eqiad.wmflabs.
2018-03-16T14:02:42Z mwopenstackclients.DnsManager INFO    : Updating s3.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:17Z mwopenstackclients.DnsManager INFO    : Updating s2.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:23Z mwopenstackclients.DnsManager INFO    : Updating s1.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:24Z mwopenstackclients.DnsManager INFO    : Updating s7.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:28Z mwopenstackclients.DnsManager INFO    : Updating s6.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:29Z mwopenstackclients.DnsManager INFO    : Updating s5.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:30Z mwopenstackclients.DnsManager INFO    : Updating s4.web.db.svc.eqiad.wmflabs.

This step is done by the WMCS team.

TODO: elaborate info on what is this doing

Step 9: all is done

All is done, wiki-replica contains a mirror of the production database. Finally WCMS users/tools/projects are able to query the database/tables.

This is usually done by using the sql wrapper script.

TODO: the benefits of using the sql script.

Admin guide

Docs to perform common tasks related to this service. As detailed as possible.

Who admins what

  • main productions cluster: main operations team, DBAs
  • sanitarium cluster: main operations team, DBAs
  • wiki-replicas cluster: WMCS handles most hardware, end-user and view-level operations, Data Persistence (DBAs) handles database-level and replication matters
  • wiki-replicas DNS: WMCS

Adding a wiki

User connection limits

There are two important aspects to this.

  1. The review process, which is that requests for expanding the number of connections per tool be discussed at the weekly WMCS planning meeting for approval with a requirement of a +1 from a member of the DBA team after WMCS approval. This should be a restrictive process that errs on the side of denial. Only tools with a pretty good reason and significant user base should use additional connections beyond the default limit of 10.
  2. Keeping a record of the change in modules/role/templates/mariadb/grants/wiki-replicas.sql. In the file as of this writing, line 38 provides a model for a comment (including the Phabricator task number and user) as well as the exact command that is run on the replicas. For consistency, the command should be run on all four replica hosts.

The process after approval is:

  • Determine the mysql username of the tool, which should be in any approved request. This will be something similar to s52788 and not something like tools.flubber. It can be found in the tool's $HOME/replica.my.cnf.
  • The clinic-duty person logs into the root mysql shell for each replica and runs GRANT USAGE ON *.* TO '$mysql_username'@'%' WITH MAX_USER_CONNECTIONS $number; Please note that you can always view grants on the server you are on with SHOW GRANTS FOR '$mysql_username';
  • Record the change in modules/role/templates/mariadb/grants/wiki-replicas.sql
  • To persist the change on rebuilds, add the new value to hieradata/role/common/wmcs/nfs/primary.yaml
  • Communicate with the user that they should be all set!

De-pooling Replica Servers

See Portal:Data Services/Admin/Runbooks/Depool wikireplicas

Making significant changes to the views and the indexes

View and index scripts perform DDL commands on the databases, which can cause lock contention issues both for the scripts and the users of the replicas. Problems with the views can even cause data that the view system is supposed to keep out to be exposed. These are some guidelines to prevent problems.

  • De-pool a host before doing index changes and view changes that are not simply adding a new wiki, table or database. This reduces production impact as well as fallout from mistakes.
  • If there is a concern that the data may not be in a consistent or safely-scrubbed state, find a user who is trusted (as in NDA-signed) to take a look if you aren't able to come up with some good examples yourself to go after.

Updating views

  1. Confirm with DBAs that they don't have work in progress that might interfere
  2. Merge associated gerrit change, and force puppet runs on all affected replica servers
  3. On each replica server:
    1. If modifying an existing view, these steps on each replica server
      1. Depool
        1. Write, merge, apply hiera change
        2. systemctl reload haproxy && echo "show stat" | socat /run/haproxy/haproxy.sock stdio
        3. wait for depool to take effect (check with echo "show stat" | socat /run/haproxy/haproxy.sock stdio)
      2. Kill running user processes on the labsbd* server you depooled if necessary. Processes holding locks on tables you are changing must finish or be killed to proceed or the maintain-views script will fail with a traceback about timing out waiting for a lock.
        1. Look at the processes running with a root mysql shell (sudo -i mysql --skip-ssl) by running show processlist;. If you need to be able to see all of them, and they won't show on the screen, you can use pager less to set the shell to show output in a pager.
        2. Only usernames like u##### (a user) and s##### (a tool) should be killed. Other usernames are admin or system processes. Please don't kill those. You probably only need to kill queries against tables you are changing things about, and often you can just give it an hour after depooling to let people's work finish. Sometimes you need to kill more because of subqueries or DB connection pools holding table locks with connections that don't even seem to be doing anything. Never feel bad about killing an old, inactive connection because we don't allow that anyway.
        3. If you have identified a process to kill, copy the process ID and run kill $pid;. It might not die right away, but typically, unless things are in bad shape on that replica, it will die without too much delay after closing up tables.
    2. If it is just one new table, update with: sudo maintain-views --table <affected table> --all-databases, otherwise, drop the --table argument to update all views. If this is for existing tables/views, you will want to use the --replace-all argument to avoid typing "Y" 800 times.
    3. Verify changes via the mysql commandline: sudo -i mysql --skip-ssl enwiki_p
      1. Repool
        1. Revert depooling hiera change
        2. systemctl reload haproxy
        3. Check that the original state is restored with echo "show stat" | socat /run/haproxy/haproxy.sock stdio

Update wikis location

Although it is a rare operation, sometimes production wikis get moved from one section to a different one. For example, if two wikis get moved from s3 to s5, the following steps need to taken on the wikireplicas:

  • Update modules/profile/files/openstack/base/pdns/recursor/labsdb.zone, follow this example: https://gerrit.wikimedia.org/r/619627
  • Run /usr/local/sbin/maintain-meta_p on each wikireplica host
    • Double check that the slices look good and the new wikis are on the correct slice, in this case, s5:
sql meta_p
select dbname, slice from wiki where dbname in ('muswiki', 'mhwiktionary');
  • From a cloudcontrol node (like cloudcontrol1004.wikimedia.org) run:
wmcs-wikireplica-dns --aliases --shard s3
wmcs-wikireplica-dns --aliases --shard s5


Before this new cluster, we had an old cluster composed of physical servers: labsdb1001, labsdb1002 and labsdb1003.

At the time of this writing, the last standing server is labsdb1003, which is going down in December 2017.

See also