Add a wiki

This page will walk you through how to add a new wiki project. This includes new languages on sister projects, and wikis for committees, chapters etc.

Start

• Tell in advance the Ops list, jcrespo (jynus) or Coren that this is happening so the storage layer can be prepared and checked (labs, backups, dumps). In particular, it needs to be made clear whether the wiki will be public - if so, ops will arrange for the wiki to be replicated to labs. If not, ops will need to add the wiki to $private_wikis in the puppet repository's manifests/realm.pp.

This can be fixed after the wiki creation, but it is more painful/involves custom queries.

DNS

• First of all, the DNS entry for the new wiki has to be added. Make the following changes to your local checkout of operations/dns.git and submit to gerrit for review.
  • If it is a language project, add the language code to /templates/helpers/langs.tmpl in operations/dns repo. If it is not a new language, then it has probably been done already as these are shared by all the sister projects.
  • If it is a wiki in *.wikimedia.org domain (chapter wiki or special wiki), add it to /templates/wikimedia.org. Make sure to add a mobile entry if necessary.
• Merge the change in gerrit and run authdns-update. Query the DNS servers to make sure it has been correctly deployed. See DNS#HOWTO for details.

Apache configuration

• Secondly, add Apache configuration for the new wiki by submitting it to gerrit first. They are now located at /modules/mediawiki/files/apache/sites/ in operations/puppet.git.
• Common configuration:
  • For languages projects, this step is usually not needed as it has been configured already.
  • For chapter wikis, add ServerAlias to wikimedia-chapter VirtualHost in /wikimedia.conf.
  • For Wikimania wikis, add ServerAlias to /wikimania.conf.
• After merging on Gerrit, do sync-apache and apache-graceful-all^{(needs verification?)}. See Apache for details.
• If there are any other domains which should point to the same wiki, add it to /redirects.conf and /redirects/redirects.dat.

You will firstly have to login to our deployment host. As of sometime, this is the host tin, you will have to go via bast1001. Then gather all relevant information for the new project, including all configuration variables and entries. Each wiki will have different requirements (project name, namespaces, extensions to be enabled etc).

You now have to determine if this is a new language project (say something like Spanish Wikibooks) or a chapter wiki, or a special wiki (anything else, including private wikis). Then choose the correct part of this manual:

• If it is a language project, follow steps in #Language Project)
• Else go directly to #Finish

Language Project

• Make sure the language has been approved by the Language committee at Requests for new languages on Meta-Wiki. Usually, the Maniphest task will contain a link to the approval page.
• Make the following changes on your own checkout of operations/mediawiki-config.git:
• Make sure that the language code appears in jquery.uls's langdb (file data/langdb.yaml).
• Go to #Finish

Finish

IMPORTANT: For Private Wikis

• Private wiki databases must not be replicated to the labsdb MySQL instances!
  • Before creating a database for a new private wiki, make sure to add the db name to the puppet global array $private_wikis in manifests\realm.pp.
  • Deploy this config change with puppet and manually restart the Prelabsdb-db (Sanitarium) MySQL instance that will house this wiki's db (most likely s3).
  • If you need help with this, please ask a member of the Ops team for help. This is very important.

MediaWiki configuration

• Make the following changes on your own checkout of operations/mediawiki-config.git and submit to Gerrit for review:
  • If it is a new language, add the language code (ISO 639 code: usually provided in the task) to langlist. This will be used at Special:SiteMatrix and for interwiki linking etc.
  • Update the configuration files (located at /wmf-config). Some may not be required but the following are the most common wiki-specific settings in InitialiseSettings.php:
    • Manual:$wgServer, Manual:$wgCanonicalServer, Manual:$wgLogo, Manual:$wgSitename, Manual:$wgExtraNamespaces, Manual:$wgLocaltimezone, groupOverrides.
    • Overrides for $wgServer and $wgCanonicalServer are usually not done for language projects as the default works for these.
    • Also set Manual:$wgCategoryCollation if it is needed for a language. If you are not sure, ask on the task.
    • If an extension needs to be enabled: usually done by setting a wmg variable to true and if custom configuration needs to be done for the extension, do them as well. Some extensions' configuration are located in a different file located at /wmf-config directory.
  • For *wikimedia databases, add the subdomain to the list in MWMultiVersion::setSiteInfoForWiki
• Add the wiki to the relevant dblists (under the dblists directory).

Database list	Purpose
s1.dblist s2.dblist s3.dblist s4.dblist s5.dblist s6.dblist s7.dblist	Database lists of wikis in each MySQL database cluster
all.dblist	All wikis should be listed in here
closed.dblist	Any closed (no write access, full read access) wikis
deleted.dblist	Wiki databases which MediaWiki is no longer configured to access
small.dblist medium.dblist large.dblist	Database lists of wikis arranged into their relevant size
flaggedrevs.dblist	All wikis running the FlaggedRevs extension
securepollglobal.dblist	$wgSecurePollCreateWikiGroups wikis: Board Election wikis
visualeditor-default.dblist	All wikis where VisualEditor is enabled by default
commonsuploads.dblist	All wikis which should have local uploading soft-disabled. Uploads go to Commons instead.
fishbowl.dblist	All fishbowl (restricted write access, full read access) wikis
private.dblist	All private (read and write restricted) wikis
special.dblist	All special wikis
wikidata.dblist	All wikis running the Wikidata repo
wikidataclient.dblist	All wikis running the Wikidata client (most new language-project wikis should start off like this)
wikimania.dblist	All Wikimania wikis
wikimedia.dblist	All chapter wikis
wikibooks.dblist	All Wikibooks wikis
wikinews.dblist	All Wikinews wikis
wikipedia.dblist	All Wikipedia wikis
wikiquote.dblist	All Wikiquote wikis
wikisource.dblist	All Wikisource wikis
wikiversity.dblist	All Wikiversity wikis
wikivoyage.dblist	All Wikivoyage wikis
wiktionary.dblist	All Wiktionary wikis

• Add an entry to wikiversions.json. See Deployments (one week) and mw:Deployment Train.

• Now it is time to actually create the database (this script also performs other tasks, such as notifying the 'newprojects' mailing list). This has to be done on terbium, not mira (as it's in codfw, so no DB write access). If the wiki is going to run the Wikidata client, make sure it's added to the dblist on terbium **before** running this script, or things will break. There are two different ways to do this, depending on if you did Part A, B, or C above:

• mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki languagecode projectname databasename domain
• Part A is for standard language additions to projects, like adding a Spanish Wikinews, which will be my example.

• mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki languagecode projectname databasename domain

• EG: mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki es wikinews eswikinews es.wikinews.org to add a Spanish Wikinews.

• Part B is for Chapters Wikis. My example will be adding a Finnish chapter wiki:

• mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki languagecode projectname databasename domain

• EG: mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki fi wikimedia fiwikimedia fi.wikimedia.org

• Part C is for non-standard special wikis such as committees, chapters, and the like. My example will be strategy.wikimedia.org.

• mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki languagecode projectname databasename domain

• EG: mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki en wikimedia strategywiki strategy.wikimedia.org

(Note the 'aawiki' in these examples: aawiki is in s3, so this will create the database in s3, which is usually the right place for new small wikis.)

• Merge the config change in Gerrit, and pull it onto tin
• Check *.dblist files now contain the new wiki
• Run sync-dir dblists to synchronize all database lists
• Check wikiversions.json is sane
• Run sync-wikiversions to synchronize the version number to use for this wiki
• Run sync-file wmf-config/InitialiseSettings.php
• Run sync-file w/static/images/project-logos/path-to-logo.png
• Run sync-file langlist (this must be done before the interwiki cache below)
• Unless it's a language project, update the meta:Interwiki map
• Deploy the interwiki update (for all new wiki creations):
  • mwscript extensions/WikimediaMaintenance/dumpInterwiki.php --protocolrelative > wmf-config/interwiki.php
  • Commit change, upload to gerrit for review
  • Review and merge
  • sync-file wmf-config/interwiki.php
  • Update meta:Interwiki map date of last update

RESTBase

RESTBase is a service providing a RESTful API for the projects' wikis. To enable it to serve the new wiki as well, create a patch for ops/puppet adding its domain to RESTBase's configuration.

Parsoid

Parsoid has its own copy of SiteMatrix, which needs updating using tools/fetch-sitematrix.js in mediawiki/services/parsoid.git.

Search

You shouldn't have to do much (maybe adjust shard sizes preemptively if it'll be big), the index should be automatically created by addWiki.php and all wikis now opt into Cirrus/Elastic by default. See Search/New#Adding_new_wikis for more information.

Swift

• Public wikis:
  • Create the container for thumbnails. mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php databasename --backend=local-multiwrite
• Private wikis:
  • Create the container for thumbnails. mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php databasename --backend=local-multiwrite --private

Labs DNS

Create a patch for ops/puppet to add the database to the correct list in modules/role/manifests/labs/dnsrecursor.pp

Wikidata

Only needed if the new wiki is supposed to be a Wikidata client.

In order to be able to link the new Wiki from Wikidata and have interwiki links from Wikidata to that Wiki appear on other Wikis, extensions/Wikidata/extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https needs to be run on at least all Wikidata clients (that includes Wikidata and testwikidata).

That script is known to be troublesome, you might want to ask Marius (hoo) or Katie (aude) run it for you or just create a ticket (that can be done anytime after the Wikis has been created).

Beware: The script sometimes fails with a duplicate key conflict. In that case, go to the wiki's master database and empty the sites and site_identifiers tables, then run the script again. It's probably also wise to backup these tables from Wikidata and at least one Wikipedia before running the script across the whole fleet. Breaking the sites, site_identifiers tables will break page rendering on a Wiki!

Also, make sure that the language code appears in the file client/config/WikibaseClient.default.php in the mediawiki/extensions/Wikibase repo. (Example: https://gerrit.wikimedia.org/r/#/c/288097/4.)

WikimediaMessages

Translatable project name

Add a message with the wiki name to extensions/WikimediaMessages/wikimediaprojectnames/en.json and qqq.json. The message keys should include the wiki database name (WIKI_DBNAME below) and official "human readable" name (WIKI_NAME below) as follows:

en.json

Key	Message
project-localized-name-WIKI_DBNAME	WIKI_NAME

For example, "project-localized-name-enwiki": "English Wikipedia",

qqq.json

Key	Message
project-localized-name-WIKI_DBNAME	{{ProjectNameDocumentation|url=WIKI_URL|name=WIKI_NAME|language=WIKI_LANG}}

For example, "project-localized-name-enwiki": "{{ProjectNameDocumentation|url=https://en.wikipedia.org|name=English Wikipedia|language=en}}",

Interwiki search result title

Add a message with the wiki name to extensions/WikimediaMessages/wikimedia/en.json and qqq.json in the list of search-interwiki-results messages.

Analytics

If the wiki is not private, not a wikimania, and not misc like usability/outreach/login/vote/strategy/etc., send a change proposal to analytics/refinery.git to add the wiki to static_data/pageview/whitelist/whitelist.tsv

After you finished

If there's something to import (as is often the case in new Language wikis), someone will do so, following the process described at MetaWikipedia:incubator:Incubator:Importing from Incubator (logged at MetaWikipedia:incubator:Incubator:Site creation log). Then when that's done, add the language code to the ContentTranslation registry - mediawiki/services/cxserver repository, files config.dev.yaml and config.prod.yaml, in the source and target sections in each.

After finishing, Inform the community and make a phabricator card for removing old interwikilinks and migrating them to Wikidata (For example phab:T134991 for edits such as d:Special:Diff/336584053 and w:jam:Special:Diff/12330). You can do it by yourself using interwikidata.py in pywikibot.

python pwb.py scripts/interwikidata.py -lang:LANGCODE -clean -start:! -always

See also Delete a wiki.