You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Difference between revisions of "Help:Puppet-compiler"

From Wikitech
Jump to navigation Jump to search
imported>Andrew Bogott
(→‎Catalog compiler for CloudVPS: Updated to reflect that this is now MUCH easier :))
imported>SRodlund
Line 1: Line 1:
 +
{{Cloud VPS nav}}
 +
 +
==Overview==
 +
 
You can run '''puppet-compiler''' by hand to get the results of a given puppet configuration without having to deploy it to servers.
 
You can run '''puppet-compiler''' by hand to get the results of a given puppet configuration without having to deploy it to servers.
  
Line 5: Line 9:
 
== Catalog compiler in integration Jenkins ==
 
== Catalog compiler in integration Jenkins ==
  
There is a jenkins job that takes a gerrit change and runs the compiler.
+
There is a Jenkins job that takes a Gerrit change and runs the compiler.
  
 
Steps:
 
Steps:
Line 11: Line 15:
 
# Go to https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/
 
# Go to https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/
 
# Go to "Build with parameters" https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/build
 
# Go to "Build with parameters" https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/build
# In the form, fill '''change number''' (from gerrit) and '''list of nodes'''
+
# In the form, fill '''change number''' (from Gerrit) and '''list of nodes'''
 
# Hit the '''Build''' button
 
# Hit the '''Build''' button
# Wait for the jenkins job to end
+
# Wait for the Jenkins job to end
# You can check for results in the jenkins '''Console output'''
+
# You can check for results in the Jenkins '''Console output'''
 
# You can see the compiled catalogs in a web frontend.  The URL structure is https://puppet-compiler.wmflabs.org/compiler_host/build_id where...
 
# You can see the compiled catalogs in a web frontend.  The URL structure is https://puppet-compiler.wmflabs.org/compiler_host/build_id where...
 
#* '''compiler_host''' is the hostname (without domain name) of the compiler node that Jenkins dispatched the build to.  A current list of possible compiler nodes is available at https://integration.wikimedia.org/ci/label/puppet-compiler-node/
 
#* '''compiler_host''' is the hostname (without domain name) of the compiler node that Jenkins dispatched the build to.  A current list of possible compiler nodes is available at https://integration.wikimedia.org/ci/label/puppet-compiler-node/
Line 24: Line 28:
 
== Catalog compiler for CloudVPS ==
 
== Catalog compiler for CloudVPS ==
  
The [https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/ standard jenkins-hosted catalog compiler] can now target VPS instances.  Because VMs are frequently created and deleted, it may be necessary to update the facts from whatever puppetmaster is hosting the VM in question -- instructions for doing that can be found at [[Nova Resource:Puppet-diffs]].
+
The [https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/ standard Jenkins-hosted catalog compiler] can now target VPS instances.  Because VMs are frequently created and deleted, it may be necessary to update the facts from whatever puppetmaster is hosting the VM in question -- instructions for doing that can be found at [[Nova Resource:Puppet-diffs]].
  
 
<br />
 
<br />
Line 49: Line 53:
 
* ERROR: Unable to find facts for host tools-services-01.tools.eqiad.wmflabs, skipping
 
* ERROR: Unable to find facts for host tools-services-01.tools.eqiad.wmflabs, skipping
  
If running locally, collect facts by hand from the corresponding puppetmaster. If running in the jenkins web service for a production host, follow [[Nova_Resource:Puppet-diffs#FAQ|these instructions]].
+
If running locally, collect facts by hand from the corresponding puppetmaster. If running in the Jenkins web service for a production host, follow [[Nova_Resource:Puppet-diffs#FAQ|these instructions]].
  
 
== Limitations ==
 
== Limitations ==
  
The puppet-compiler mechanism won't discover all the issues in the resulting catalog. If the catalog was compiled OK by jenkins, you may still find some issues when running the puppet agent.<br>
+
The puppet-compiler mechanism won't discover all the issues in the resulting catalog. If the catalog was compiled OK by Jenkins, you may still find some issues when running the puppet agent.<br>
 
Some known limitations:
 
Some known limitations:
  
 
* Files sources. When declaring a <code>File { '/my/file':</code>, the path information you specified in the <code>content</code> parameter will be resolved at puppet agent runtime.
 
* Files sources. When declaring a <code>File { '/my/file':</code>, the path information you specified in the <code>content</code> parameter will be resolved at puppet agent runtime.
* Private hiera lookups. The way hiera fetches data may vary between how is done in the puppet-compiler process to how is done in the final puppet master. Specifically, secrets in the private repo.
+
* Private hiera lookups. The way hiera fetches data may vary between how its done in the puppet-compiler process to how its done in the final puppet master. Specifically, secrets in the private repo.
 
* Hiera behaviour. Currently, we don't have a way to know in concrete how hiera is behaving when compiling the catalog. See [[phab:T215507 | phabricator ticket T215507]] for more information.
 
* Hiera behaviour. Currently, we don't have a way to know in concrete how hiera is behaving when compiling the catalog. See [[phab:T215507 | phabricator ticket T215507]] for more information.
  
'''TODO:''' I'm writting from memory. This section can be improved for accuracy.
+
<!--commenting this out for now
 +
'''TODO:''' I'm writing from memory. This section can be improved for accuracy.
 +
-->
 +
 
 +
{{:Help:Cloud Services communication}}
  
 
== See also ==
 
== See also ==
Line 70: Line 78:
 
* {{phabT|97081}}
 
* {{phabT|97081}}
  
 +
[[Category:Cloud VPS]]
 +
[[Category:Documentation]]
 +
[[Category:Cloud Services]]
 
[[Category:Puppet]]
 
[[Category:Puppet]]
[[Category:Documentation]]
 
[[Category:Cloud VPS]]
 

Revision as of 20:25, 13 February 2020


Overview

You can run puppet-compiler by hand to get the results of a given puppet configuration without having to deploy it to servers.

This page gives some instructions on doing so.

Catalog compiler in integration Jenkins

There is a Jenkins job that takes a Gerrit change and runs the compiler.

Steps:

  1. Push your change to gerrit using git-review
  2. Go to https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/
  3. Go to "Build with parameters" https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/build
  4. In the form, fill change number (from Gerrit) and list of nodes
  5. Hit the Build button
  6. Wait for the Jenkins job to end
  7. You can check for results in the Jenkins Console output
  8. You can see the compiled catalogs in a web frontend. The URL structure is https://puppet-compiler.wmflabs.org/compiler_host/build_id where...
    • compiler_host is the hostname (without domain name) of the compiler node that Jenkins dispatched the build to. A current list of possible compiler nodes is available at https://integration.wikimedia.org/ci/label/puppet-compiler-node/
    • build_id is the unique id of the Jenkins build (changes with every run).
    • This link is automatically constructed and can be found at the bottom of the Jenkins console output after each build.

NOTE: this method won't work for CloudVPS instances (see next method below).

Catalog compiler for CloudVPS

The standard Jenkins-hosted catalog compiler can now target VPS instances. Because VMs are frequently created and deleted, it may be necessary to update the facts from whatever puppetmaster is hosting the VM in question -- instructions for doing that can be found at Nova Resource:Puppet-diffs.


Catalog compiler local run (pcc utility)

There is a also a tool called pcc under the operations/puppet/utils repo. You'll need your Jenkins API token to make it work, retrievable under https://integration.wikimedia.org/ci/user/$YOURUSERNAME/configure.

Example:

$ ./utils/pcc GERRIT_CHANGE_NUMBER LIST_OF_NODES --username YOUR_USERNAME --api-token 12312312312312313  
$ ./utils/pcc 282936 oxygen.eqiad.wmnet --username batman --api-token 12312312312312313

Troubleshooting

Some common errors and mistakes.

  • Catalog for Cloud VPS instances doesn't get any classes/roles.

This happens because $::realm is not set to labs. There are patches in place to fix this, but the puppet-compiler software needs to be released with these patches.

  • ERROR: Unable to find facts for host tools-services-01.tools.eqiad.wmflabs, skipping

If running locally, collect facts by hand from the corresponding puppetmaster. If running in the Jenkins web service for a production host, follow these instructions.

Limitations

The puppet-compiler mechanism won't discover all the issues in the resulting catalog. If the catalog was compiled OK by Jenkins, you may still find some issues when running the puppet agent.
Some known limitations:

  • Files sources. When declaring a File { '/my/file':, the path information you specified in the content parameter will be resolved at puppet agent runtime.
  • Private hiera lookups. The way hiera fetches data may vary between how its done in the puppet-compiler process to how its done in the final puppet master. Specifically, secrets in the private repo.
  • Hiera behaviour. Currently, we don't have a way to know in concrete how hiera is behaving when compiling the catalog. See phabricator ticket T215507 for more information.


Communication and support

We communicate and provide support through several primary channels. Please reach out with questions and to join the conversation.

Communicate with us
Connect Best for
Phabricator Workboard #Cloud-Services Task tracking and bug reporting
IRC Channel #wikimedia-cloud connect General discussion and support
Mailing List cloud@ Information about ongoing initiatives, general discussion and support
Announcement emails cloud-announce@ Information about critical changes (all messages mirrored to cloud@)
News wiki page News Information about major near-term plans
Blog Clouds & Unicorns Learning more details about some of our work

See also