You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Nova Resource:Puppet-diffs/Documentation: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Dzahn
(replace old compiler1001-style names with new pcc-worker-style names in all examples)
imported>Dzahn
(add section how to handle 'out of disk')
 
Line 87: Line 87:
sudo -u jenkins-deploy git -C /var/lib/catalog-differ/private pull
sudo -u jenkins-deploy git -C /var/lib/catalog-differ/private pull
</code>
</code>
==== Out of disk space ====
Sometimes, especially when multiple users compiled a change on all hosts (" / leaving the host form field empty), the compiler VMs can run out of disk space. Then nobody can compile until some old data is deleted. Find the right compiler instance name from the error message and then, for example:
ssh pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud
cd /srv/jenkins/puppet-compiler/output
du -hs * 
identify largest builds, for example "du -hs * | grep G" to find only those that are over 1G. A common run uses maybe 50MB, a run on * uses 5GB!
on https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/ find the matching number and click "delete build" in the web UI
rm -rf <directory name> in the file system

Latest revision as of 21:50, 8 September 2022

Nova Resource:Puppet-diffs/Documentation

Description

Puppet-compiler simulates puppet changes and determine the effective difference before/after a given change to puppet files.

This project contains execution hosts for the Jenkins job operations-puppet-catalog-compiler. It computes the effective difference before/after a proposed puppet change for a given set of nodes. Also known as puppet compiler.

Build output is accessible via web service http://puppet-compiler.wmflabs.org/

For documentation on the service, see Puppet Testing.

FAQ

Who has access?

Results of completed compiler jobs are published for all to see. The UI for launching custom jobs can be operated by people in the 'wmf' ldap group and also probably by members of the 'nda' and 'wmde' groups.

Regular users can also schedule puppet compiler jobs for a patch by specifying hostnames in the git patch description, e.g.:

Phabricator: Fix aphlict to not try and start service if ensure == absent

Hosts: phab1003.eqiad.wmnet, phab2001.codfw.wmnet

Change-Id: Id899bdc35e203fb620d4bce6b426b2c2b93dd9ff

How to update the compiler's facts? (e.g. INFO: Unable to find facts for host conf2001.codfw.wmnet, skipping)

You'll need: 1)access to all the puppet master workers (puppetmaster::servers in hieradata) 2) access to the compiler hosts (membership to the project) 3) have ruby and ruby-safe-yaml packages installed 4) a local checkout of operations/puppet git tree. Then launch this script from your local checkout of the puppet repository for each compiler hosts. The list of compiler hosts is available in Jenkis. As of Dec. 20th 2018 there are only two compiler hosts and one is the default for the script, so you just have to run:

# Run this for all the compiler hosts (see https://integration.wikimedia.org/ci/label/puppet-compiler-node/ for the current list of compilers)
PUPPET_COMPILER=<fqdn-of-compiler-host> ./modules/puppet_compiler/files/compiler-update-facts

# Example bash one-liner to be run from the root of operations/puppet git on a laptop/workstation with root access to the puppet masters
COMPILERS="pcc-worker1001.puppet-diffs.eqiad1.wikimedia.cloud pcc-worker1002.puppet-diffs.eqiad1.wikimedia.cloud pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud"; for COMPILER in $COMPILERS; do PUPPET_COMPILER="$COMPILER" ./modules/puppet_compiler/files/compiler-update-facts; done

It will cycle through all the puppet master workers and sync the facts from all of them

Only the most recent fact for each host will be kept on the compiler host.

FYI: Jcrespo got a warning after running the above:

/usr/lib/python3/dist-packages/urllib3/connection.py:337: SubjectAltNameWarning: Certificate for puppetdb1001.eqiad.wmnet has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)

SubjectAltNameWarning

How to update the facts for cloud VMs? (e.g. INFO: Unable to find facts for host util-abogott-stretch.testlabs.eqiad.wmflabs, skipping)

Different Cloud VPS VMs use different puppet masters, and you can specify which master to use with the PUPPET_MASTER environment variable. To refresh the facts for a given VM, first determine which puppet master it uses. For example, refreshing for the default cloud puppetmaster looks like this:

COMPILERS="pcc-worker1001.puppet-diffs.eqiad1.wikimedia.cloud pcc-worker1002.puppet-diffs.eqiad1.wikimedia.cloud pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud"; for COMPILER in $COMPILERS; do PUPPET_MASTER=cloud-puppetmaster-03.cloudinfra.eqiad.wmflabs PUPPET_COMPILER="$COMPILER" ./modules/puppet_compiler/files/compiler-update-facts; done

To update facts for other WMCS projects:

Toolforge

COMPILERS="pcc-worker1001.puppet-diffs.eqiad1.wikimedia.cloud pcc-worker1002.puppet-diffs.eqiad1.wikimedia.cloud pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud"; for COMPILER in $COMPILERS; do PUPPET_MASTER=tools-puppetmaster-02.tools.eqiad.wmflabs PUPPET_COMPILER="$COMPILER" ./modules/puppet_compiler/files/compiler-update-facts; done

Integration

COMPILERS="pcc-worker1001.puppet-diffs.eqiad1.wikimedia.cloud pcc-worker1002.puppet-diffs.eqiad1.wikimedia.cloud pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud"; for COMPILER in $COMPILERS; do PUPPET_MASTER=integration-puppetmaster-02.integration.eqiad1.wikimedia.cloud PUPPET_COMPILER="$COMPILER" ./modules/puppet_compiler/files/compiler-update-facts; done

deployment-prep

COMPILERS="pcc-worker1001.puppet-diffs.eqiad1.wikimedia.cloud pcc-worker1002.puppet-diffs.eqiad1.wikimedia.cloud pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud"; for COMPILER in $COMPILERS; do PUPPET_MASTER=deployment-puppetmaster04.deployment-prep.eqiad1.wikimedia.cloud PUPPET_COMPILER="$COMPILER" ./modules/puppet_compiler/files/compiler-update-facts; done

The puppet compiler stores facts for each puppet master separately. If there are multiple fact files for a given FQDN (for example due to a change in puppetmaster) it will use which ever fact file was most-recently.

How do you run the puppet-compiler locally on a compiler host?

From time to time it may be necessary to attempt compilation from a shell on one of the compilers. In order to do so, first ensure you have shell access to the puppet-diffs openstack instances. Ask a horizon project admin to add you to the project if you are unable to log in. After you've logged in to a compiler host via ssh:

# Become the jenkins-deploy user
sudo su - jenkins-deploy

# Run the puppet-compiler (optionally you may add --debug to the end of the command for additional debugging output)
CHANGE=<gerrit change number> NODES=<comma separated list of fqdns to compile> BUILD_NUMBER=<unique build number> puppet-compiler

Update secrets / labs/private

The compiler uses fake secrets from the public repository labs/private.git. To update it, on each of the compilers: sudo -u jenkins-deploy git -C /var/lib/catalog-differ/private pull

Out of disk space

Sometimes, especially when multiple users compiled a change on all hosts (" / leaving the host form field empty), the compiler VMs can run out of disk space. Then nobody can compile until some old data is deleted. Find the right compiler instance name from the error message and then, for example:

ssh pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud
cd /srv/jenkins/puppet-compiler/output
du -hs *  
identify largest builds, for example "du -hs * | grep G" to find only those that are over 1G. A common run uses maybe 50MB, a run on * uses 5GB!
on https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/ find the matching number and click "delete build" in the web UI
rm -rf <directory name> in the file system