You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Add a wiki

From Wikitech-static
Revision as of 00:17, 3 February 2017 by imported>Dereckson (→‎Database creation: sync-file alias has been removed)
Jump to navigation Jump to search

This page documents the process for adding a new wiki. This includes new languages on sister projects, and wikis for committees, chapters etc.

Preparation

The following steps need to be completed before the wiki database may be created.

Notify

  • Create a Phabricator task for the wiki creation (if one doesn't exist already).
    • Ensure T18976 is the parent task.
    • It should be tagged with "Wikimedia-Site-requests".
  • Create (if not already) a sub task "Prepare and check storage layer for <new wiki>".
    • It should be tagged with "DBA" and "Labs".
  • Notify the Operations list. In particular, it needs to be made clear whether the wiki should be public, or private. If public, ops will arrange for the wiki to be replicated to labs. If private, ops will need to add the wiki to $private_wikis in operations/puppet.git:/manifests/realm.pp.

DNS

First of all, ensure the relevant domain names exist for the new wiki. Make the following changes in a commit for the operations/dns.git repo and submit to Gerrit for review.

  • If it is a language project, ensure the language code is present in /templates/helpers/langs.tmpl. This is shared between all sister projects. If another sister project has the same language already, then this has probably been done already.
  • If it is a subdomain of ".wikimedia.org" domain (chapter wiki or special wiki)
  • Merge the change in Gerrit and run authdns-update.
  • Query the DNS servers to make sure it has been correctly deployed. See DNS#HOWTO for details.
  • For new languages, there is also a need to regenerate zones. Run on ns0, ns1 and ns2: authdns-gen-zones -f /srv/authdns/git/templates /etc/gdnsd/zones && gdnsd checkconf && gdnsd reload-zones

Apache configuration

Apache configuration is located at /modules/mediawiki/files/apache/sites/ in the operations/puppet.git repo.

  • Common configuration:
    • For a new language project, this step is usually not needed as shared configuration already covers it.
    • For a new chapter wiki, add a ServerAlias to the "wikimedia-chapter" virtual host in /wikimedia.conf.
    • For Wikimania wikis, add a ServerAlias to /wikimania.conf.
  • After the change is merged in Gerrit, deploy the configuration change and (if needed) gracefully restart app servers. See Apache# Deploying config for details.
  • If there are additional domains that should point to the same wiki, add it to /redirects/redirects.dat and regenerate /redirects.conf.

Language committee

Determine if this is a new language project or chapter wiki (say something like Spanish Wikibooks), or something else (special wiki or private wiki).

For a new language project, follow the steps below. For all other wikis, go directly to #Install.

Install

IMPORTANT: For Private Wikis

  • Private wiki databases must not be replicated to the Labs DB MySQL instances!
    • Before creating a database for a private wiki, make sure to add the db name to the puppet global array $private_wikis in operations/puppet.git:/manifests/realm.pp.
    • Deploy this config change with puppet and manually restart the Prelabsdb-db MySQL instance (Sanitarium) on the server that will house this wiki's db (most likely s3).
    • If you need help with this, please ask a member of the Ops team for help. This is very important.

MediaWiki configuration

Gather all relevant information about the new project. Each wiki has different requirements (project name, namespaces, extensions to be enabled etc).

Make the following changes on your own checkout of operations/mediawiki-config.git and submit to Gerrit for review:

  • Add the wiki to the relevant dblists (in the dblists directory).
Database list Purpose
s1.dblist
s2.dblist
s3.dblist
s4.dblist
s5.dblist
s6.dblist
s7.dblist
Every wiki must be in one of these.

Database lists of wikis in each MySQL database cluster

all.dblist All wikis must be listed here.
closed.dblist Any closed (no write access, full read access) wikis
deleted.dblist Wiki databases which MediaWiki is no longer configured to access
small.dblist
medium.dblist
large.dblist
Every wiki must be in one of these.

Database lists of wikis arranged into their relevant size.

flaggedrevs.dblist All wikis running the FlaggedRevs extension
securepollglobal.dblist $wgSecurePollCreateWikiGroups wikis: Board Election wikis
visualeditor-default.dblist All wikis where VisualEditor is enabled by default
commonsuploads.dblist All wikis which should have local uploading soft-disabled. Uploads go to Commons instead.
fishbowl.dblist All fishbowl (restricted write access, full read access) wikis
private.dblist All private (read and write restricted) wikis
wikidata.dblist All wikis running the Wikidata repo
wikidataclient.dblist All wikis running the Wikidata client (most new language-project wikis should start off like this)
wikimania.dblist

wikimedia.dblist wikibooks.dblist

wikinews.dblist wikipedia.dblist

wikiquote.dblist wikisource.dblist

wikiversity.dblist wikivoyage.dblist

wiktionary.dblist special.dblist

Every wiki must be in one of these.

Sister project, Wikimania, chapter, or special.

NOTE: Some wikis maybe in special and one other list.

Database creation

Once the above is reviewed, merged and pull down on the deployment host also pull it down on terbium (scap pull), but do not yet deploy to main app servers until after the database is created.

Now it's time to actually create the database. The installation script also performs other tasks, such as notifying the "newprojects" mailing list. The installation script must be run from terbium (not the deployment host). If the wiki is going to be a Wikidata client, make sure it's present in the wikidataclient.dblist file and that the new version of that file has been pulled down on terbium **before** running the installation script, or things will break.

The syntax is as follows:

mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki <languagecode> <projectname> <databasename> <domain>

  • For a new language projects - for example a Spanish Wikinews:
    mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki es wikinews eswikinews es.wikinews.org
  • For a new chapters wikis - for example a Finnish chapter wiki:
    mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki fi wikimedia fiwikimedia fi.wikimedia.org
  • For non-standard special wikis (such as committees, or unique projects like meta or commons) - for example strategy.wikimedia.org:
    mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki en wikimedia strategywiki strategy.wikimedia.org

Note: In these examples 'aawiki' is used because it is a wiki on s3 - which means the new wiki will also be created on s3. This is usually the right place for new small wikis.

  1. Merge the config change in Gerrit, and pull it onto tin.
  2. Verify that the *.dblist files now contain the new wiki.
  3. Run scap sync-dir dblists to synchronize all database lists
  4. Verify wikiversions.json is sane.
  5. Run scap sync-wikiversions to synchronize the version number to use for this wiki
  6. Run scap sync-file wmf-config/InitialiseSettings.php
  7. Run scap sync-dir w/static/images/project-logos/
  8. Run scap sync-file langlist (this must be done before deploying the updated interwiki cache - step 10)
  9. Unless it's a language project, add the project to the meta:Interwiki map.
  10. Regenerate the interwiki cache and deploy it. (For all new wikis, not just new language projects.)
    • mwscript extensions/WikimediaMaintenance/dumpInterwiki.php --protocolrelative > wmf-config/interwiki.php
    • Commit change, upload to Gerrit for review
    • Review and merge.
    • scap sync-file wmf-config/interwiki.php
    • Edit meta:Interwiki map and update when the last update was.

RESTBase

RESTBase is a service providing a RESTful API for the projects' wikis. To enable it to serve the new wiki as well, create a patch for operations/puppet adding its domain to RESTBase's configuration (restbase/templates/config.yaml.erb). Add Mobrovac to review the patch, and get his +1 before contacting ops (a good ops contact for this service is Filippo Giunchedi) to get it merged. This will then require a restart of the RB service across the cluster, which can be done by ops or restbase-admins/restbase-roots.

Parsoid

Parsoid has its own copy of SiteMatrix, which needs updating using tools/fetch-sitematrix.js in mediawiki/services/parsoid.git.

Then, once merged, it must be deployed; if not, loading VisualEditor will fail with errors such as "Failed to load resource: the server responded with a status of 404".

Search

You shouldn't have to do much (maybe adjust shard sizes preemptively if it'll be big), the index should be automatically created by addWiki.php and all wikis now opt into Cirrus/Elastic by default. See Search/New#Adding new wikis for more information.

Swift

  • Public wikis:
    • Create the container for thumbnails. mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php databasename --backend=local-multiwrite
  • Private wikis:
    • Create the container for thumbnails. mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php databasename --backend=local-multiwrite --private

Labs

  • Create a patch for operations/puppet to add the database to the correct list in modules/role/manifests/labs/dnsrecursor.pp
  • Once it is replicating to the labsdb servers:
    • Run maintain-views --databases $wiki --debug on each
    • Insert a new row in meta_p.wiki for the new wiki.

Post-install

Wikidata

Only needed if the new wiki is supposed to be a Wikidata client.

In order to be able to link the new wiki from Wikidata, and to allow interwiki links from Wikidata to the new wiki, run extensions/Wikidata/extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https on at least all Wikidata clients (including wikidatawiki itself and testwikidata).

That script is known to be troublesome, you might want to ask Marius (hoo) or Katie (aude) run it for you or just create a ticket (that may be done anytime after the wiki was created).

Beware: The script sometimes fails with a duplicate key conflict. In that case, go to the wiki's master database and empty the sites and site_identifiers tables, then run the script again. It's probably also wise to backup these tables from Wikidata and at least one Wikipedia before running the script across the whole fleet. Breaking the sites, site_identifiers tables will break page rendering of many wikis!

Make sure that the language code appears in the file client/config/WikibaseClient.default.php in the mediawiki/extensions/Wikibase repo. (Example: https://gerrit.wikimedia.org/r/288097.)

WikimediaMessages

Translatable project name

Add a message with the wiki name to extensions/WikimediaMessages/i18n/wikimediaprojectnames/en.json and qqq.json. The message keys should include the wiki database name (WIKI_DBNAME below) and official "human readable" name (WIKI_NAME below) as follows:

en.json
Key Message
project-localized-name-WIKI_DBNAME WIKI_NAME

For example, "project-localized-name-enwiki": "English Wikipedia",

qqq.json
Key Message
project-localized-name-WIKI_DBNAME {{ProjectNameDocumentation|url=WIKI_URL|name=WIKI_NAME|language=WIKI_LANG}}

For example, "project-localized-name-enwiki": "{{ProjectNameDocumentation|url=https://en.wikipedia.org|name=English Wikipedia|language=en}}",

Interwiki search result title

Add a message with the wiki name to extensions/WikimediaMessages/i18n/wikimediainterwikisearchresults/en.json and qqq.json in the list of search-interwiki-results messages.

Analytics

If the wiki is not private, not a Wikimania conference wiki, and not special wiki like usability/outreach/login/vote/strategy/etc., send a change proposal to analytics/refinery.git that adds the wiki to static_data/pageview/whitelist/whitelist.tsv

Incubator

If there's something to import (as is often the case in new language projects), someone will do so. Their process is described at Incubator:Importing from Incubator (logged at Incubator:Site creation log).

cxserver

For new Wikipedia projects only: Add the language code to the ContentTranslation registry - mediawiki/services/cxserver repository, files registry.yaml (included by config.dev.yaml) and registry.wikimedia.yaml (included by config.prod.yaml), in the source and target sections in each.

Once merged to master, ping the project to deploy the change. That requires to sync repositories, ie to update mediawiki/services/cxserver/deploy to match mediawiki/services/cxserver. See https://gerrit.wikimedia.org/r/#/c/303763/ for an example commit.

Clean up interwiki links

After any import Incubator is completed, Inform the community and make a Phabricator task for removing old interwiki links and migrating them to Wikidata (For example T134991 for edits such as d:Special:Diff/336584053 and jam:Special:Diff/12330). You can do it by yourself using interwikidata.py in pywikibot.

python pwb.py scripts/interwikidata.py -lang:LANGCODE -clean -start:! -always

Tell wikistats labs project to add the wiki

Create a Phabricator task (subtask of the main task to create the wiki) with the tag "Labs-project-wikistats" and just ask for the wiki to be added. (What needs to be done can be seen f.e. in https://phabricator.wikimedia.org/T140970#2531977)

See also