You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Doc.wikimedia.org: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Slavina Stefanova
m (→‎Microsites: Fix typo)
imported>Krinkle
Line 2: Line 2:


== Service ==
== Service ==
Currently hosted on [[doc1001]] (through [[misc-web-lb]]). Formerly on [[gallium]].
Currently hosted on doc1002 and doc2001 (through [[misc-web-lb]]). Web traffic routing is directed via <code>doc.discovery.wmnet</code>. The same DNS discovery name is also used as the destination of the "publish-to-doc" Jenkins job.
 
Prior to 2018, the doc site was previously hosted on [[gallium]].


== Architecture ==
== Architecture ==
Line 37: Line 39:
Each microsite is generated by a post-merge Jenkins CI job, which executes on a Jenkins CI agent. Each Jenkins CI agent is a WMCS instance in the <code>[[Nova Resource:Integration|integration]]</code> project. These execute untrusted containers, packages, and user-submitted code and are not allowed to communicate with production hosts (such as the doc.wikimedia.org host).
Each microsite is generated by a post-merge Jenkins CI job, which executes on a Jenkins CI agent. Each Jenkins CI agent is a WMCS instance in the <code>[[Nova Resource:Integration|integration]]</code> project. These execute untrusted containers, packages, and user-submitted code and are not allowed to communicate with production hosts (such as the doc.wikimedia.org host).


After a microsite is generated, the <code>publish-to-doc</code> job executes on the primary Jenkins host in production, which pulls the microsite from the job workspace on the CI agent where the post-merge job ran (using rsync and ssh). This is then relayed further using rsync to on <code>doc.discovery.wmnet</code> (<code>rsync://doc.discovery.wmnet/doc/</code>). The DNS entry is an alias to the machine serving the documentations (<code>doc1001.eqiad.wmnet</code> as of November 2021).
After a microsite is generated, the <code>publish-to-doc</code> job executes on the primary Jenkins host in production, which pulls the microsite from the job workspace on the CI agent where the post-merge job ran (using rsync and ssh). This is then relayed further using rsync to <code>doc.discovery.wmnet</code> (<code>rsync://doc.discovery.wmnet/doc/</code>).


== Runbook ==
== Runbook ==

Revision as of 23:47, 1 July 2022

doc.wikimedia.org is the publication target for auto-generated software documentation.

Service

Currently hosted on doc1002 and doc2001 (through misc-web-lb). Web traffic routing is directed via doc.discovery.wmnet. The same DNS discovery name is also used as the destination of the "publish-to-doc" Jenkins job.

Prior to 2018, the doc site was previously hosted on gallium.

Architecture

Landing pages

The navigational structure and page layout is server-side generated by simple PHP script. This provides navigation header, footer, and page layout. The main content of each page comes from either an HTML file, YAML file, or PHP function that produces a directory index.

The root page lists project for which documentation is open, one can see it as a catalogue of some of our open source product. Items can be added by editing the file org/wikimedia/doc/opensource.yaml. See the repository README.md for additional details and to test your change locally.

Examples of URLs in scope:

Microsites

The landing pages provide discovery of the microsites for API documentation, demos, and coverage reports. Each microsite is standalone, statically generated, and individually deployed. They are published by post-merge Jenkins CI jobs into the /srv/doc/ directory outside the landing pages' document root.

Examples of URLs in scope:

Publishing

Each microsite is generated by a post-merge Jenkins CI job, which executes on a Jenkins CI agent. Each Jenkins CI agent is a WMCS instance in the integration project. These execute untrusted containers, packages, and user-submitted code and are not allowed to communicate with production hosts (such as the doc.wikimedia.org host).

After a microsite is generated, the publish-to-doc job executes on the primary Jenkins host in production, which pulls the microsite from the job workspace on the CI agent where the post-merge job ran (using rsync and ssh). This is then relayed further using rsync to doc.discovery.wmnet (rsync://doc.discovery.wmnet/doc/).

Runbook

Switch primary host

Pre-switch checklist:

  • The new host must be provisioned with the doc role.
  • For the landing site:
    • The new host must be a scap/dsh target for ci-docroot, and initialised after the last deployment.
    • Else do at least one dummy "scap deploy" after adding it to the targets to ensure it has the latest version.
  • For the microsites content:
    • The new host must be included in Puppet profile::doc::all_hosts, which ensures the current primary rsyncs its content first.

Actual switch:

  1. Prepare a DNS change to set doc.discovery.wmnet to the new host. This will control where new content is synced to by Jenkins, and which backend external HTTP traffic is routed to.
  2. Prepare a Puppet change to set profile::doc::active_host to the new host. This will toggle the hourly cronjob that rsyncs microsites content from the primary to secondary hosts. It is important to run Puppet agent on all hosts after merging this so as to not have a 30-minute window during which Jenkins may be syncing to a read-only server.
  3. Deploy the DNS change, quickly followed by the Puppet merge and run.

If a post-merge Jenkins job happens to be running around this time, or if there are reports of lost content, a CI admin can re-run the last post-merge job for their software documentation (in Jenkins, or via Zuul) to generate and sync it a second time after the switch.

Deploy landing page

The "landing page" part of this service comes from the integration/docroot Git repository, and is deployed from a deployment server using Scap:

 ssh deployment.eqiad.wmnet
 cd /srv/deployment/integration/docroot
 git fetch origin
 # review the diff
 git rebase
 scap deploy 'My Change-Id and/or other explanation'

External link

See also