You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Doc.wikimedia.org: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Hashar
(Explain how one can amend the main page)
imported>Krinkle
(→‎Runbooks: Document procedure after phab:T247653)
Line 20: Line 20:


=== Microsites ===
=== Microsites ===
These landing pages link out to microsites with API documentation, demos, and coverage reports. Each of these is entirely standalone and statically generated. The are published by post-merge CI jobs into the <code>/srv/doc/</code> directory outside the landing pages' document root.
The landing pages provide discovery of the microsites for API documentation, demos, and coverage reports. Each microsite is standalone, statically generated, and individually deployed. The are published by post-merge Jenkins CI jobs into the <code>/srv/doc/</code> directory outside the landing pages' document root.
 
To learn how we (securely) transfer these sites from CI to the doc.wikimedia.org server, see [[mw:Continuous integration/Documentation generation|Continuous integration/Documentation generation]] on mediawiki.org.


Examples of URLs in scope:
Examples of URLs in scope:
Line 34: Line 32:
* https://doc.wikimedia.org/cover/mediawiki-libs-Minify/ (PHPUnit test coverage).
* https://doc.wikimedia.org/cover/mediawiki-libs-Minify/ (PHPUnit test coverage).
* https://doc.wikimedia.org/cover/unicodejs/ (Istanbul.JS test coverage).
* https://doc.wikimedia.org/cover/unicodejs/ (Istanbul.JS test coverage).
=== Publishing ===
{{See also|mw:Continuous integration/Documentation generation}}
Each microsite is generated by a post-merge Jenkins CI job, which executes on a Jenkins CI agent. Each Jenkins CI agent is a WMCS instance in the <code>[[Nova Resource:Integration|integration]]</code> project. These execute untrusted containers, packages, and user-submitted code and are not allowed to communicate with production hosts (such as the doc.wikimedia.org host).
After a microsite is generated, the <code>publish-to-doc</code> job executes on the primary Jenkins host in production, which pulls the microsite from the job workspace on the CI agent where the post-merge job ran (using rsync and ssh). This is then relayed further using rsync to on <code>doc.discovery.wmnet</code> (<code>rsync://doc.discovery.wmnet/doc/</code>). The DNS entry is an alias to the machine serving the documentations (<code>doc1001.eqiad.wmnet</code> as of November 2021).
== Runbook ==
=== Switch primary host ===
Pre-switch checklist:
* The new host must be provisioned with the <code>profile::doc</code> role.
* For the landing site:
** The new host must be a scap/dsh target for <code>ci-docroot</code>, and initialised after the last deployment.
** Else do at least one dummy "scap deploy" after adding it to the targets to ensure it has the latest version.
* For the microsites content:
** The new host must be included in Puppet <code>profile::doc::all_hosts</code>, which ensures the current primary rsyncs its content first.
Actual switch:
# Prepare a DNS change to set <code>doc.discovery.wmnet</code> to the new host. This will control where new content is synced to by Jenkins, and which backend external HTTP traffic is routed to.
# Prepare a Puppet change to set <code>profile::doc::active_host</code> to the new host. This will toggle the hourly cronjob that rsyncs microsites content from the primary to secondary hosts. It is important to '''run Puppet agent on all hosts after merging this''' so as to not have a 30-minute window during which Jenkins may be syncing to server where the cronjob will be '''deleting''' unexpected content.
# Deploy the DNS change, quickly followed by the Puppet merge and run.
If a post-merge Jenkins job happens to be running around this time, or if there are reports of lost content, a CI admin can re-run the last post-merge job for their software documentation (in Jenkins, or via Zuul) to generate and sync it a second time after the switch.
=== Deploy landing page ===
The "landing page" part of this service comes from the [[git:integration/docroot|integration/docroot]] Git repository, and is deployed from a [[deployment server]] using [[Scap]]:
  ssh deployment.eqiad.wmnet
  cd /srv/deployment/integration/docroot
  git fetch origin
  # review the diff
  git rebase
  scap deploy 'My Change-Id and/or other explanation'


== External link ==
== External link ==
Line 43: Line 76:
{{lowercase}}
{{lowercase}}
[[Category:Services]]
[[Category:Services]]
[[Category:Runbooks]]

Revision as of 23:18, 30 November 2021

doc.wikimedia.org is the publication target for auto-generated software documentation.

Service

Currently hosted on doc1001 (through misc-web-lb). Formerly on gallium.

Architecture

Landing pages

The navigational structure and page layout is server-side generated by simple PHP script. This provides navigation header, footer, and page layout. The main content of each page comes from either an HTML file, YAML file, or PHP function that produces a directory index.

The root page lists project for which documentation is open, one can see it as a catalogue of some of our open source product. Items can be added by editing the file org/wikimedia/doc/opensource.yaml. See the repository README.md for additional details and to test your change locally.

Examples of URLs in scope:

Microsites

The landing pages provide discovery of the microsites for API documentation, demos, and coverage reports. Each microsite is standalone, statically generated, and individually deployed. The are published by post-merge Jenkins CI jobs into the /srv/doc/ directory outside the landing pages' document root.

Examples of URLs in scope:

Publishing

Each microsite is generated by a post-merge Jenkins CI job, which executes on a Jenkins CI agent. Each Jenkins CI agent is a WMCS instance in the integration project. These execute untrusted containers, packages, and user-submitted code and are not allowed to communicate with production hosts (such as the doc.wikimedia.org host).

After a microsite is generated, the publish-to-doc job executes on the primary Jenkins host in production, which pulls the microsite from the job workspace on the CI agent where the post-merge job ran (using rsync and ssh). This is then relayed further using rsync to on doc.discovery.wmnet (rsync://doc.discovery.wmnet/doc/). The DNS entry is an alias to the machine serving the documentations (doc1001.eqiad.wmnet as of November 2021).

Runbook

Switch primary host

Pre-switch checklist:

  • The new host must be provisioned with the profile::doc role.
  • For the landing site:
    • The new host must be a scap/dsh target for ci-docroot, and initialised after the last deployment.
    • Else do at least one dummy "scap deploy" after adding it to the targets to ensure it has the latest version.
  • For the microsites content:
    • The new host must be included in Puppet profile::doc::all_hosts, which ensures the current primary rsyncs its content first.

Actual switch:

  1. Prepare a DNS change to set doc.discovery.wmnet to the new host. This will control where new content is synced to by Jenkins, and which backend external HTTP traffic is routed to.
  2. Prepare a Puppet change to set profile::doc::active_host to the new host. This will toggle the hourly cronjob that rsyncs microsites content from the primary to secondary hosts. It is important to run Puppet agent on all hosts after merging this so as to not have a 30-minute window during which Jenkins may be syncing to server where the cronjob will be deleting unexpected content.
  3. Deploy the DNS change, quickly followed by the Puppet merge and run.

If a post-merge Jenkins job happens to be running around this time, or if there are reports of lost content, a CI admin can re-run the last post-merge job for their software documentation (in Jenkins, or via Zuul) to generate and sync it a second time after the switch.

Deploy landing page

The "landing page" part of this service comes from the integration/docroot Git repository, and is deployed from a deployment server using Scap:

 ssh deployment.eqiad.wmnet
 cd /srv/deployment/integration/docroot
 git fetch origin
 # review the diff
 git rebase
 scap deploy 'My Change-Id and/or other explanation'

External link

See also