You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Help:Puppet-compiler: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Arturo Borrero Gonzalez
(→‎Troubleshooting: remove link to Cloud VPS)
imported>BryanDavis
(Formatting, small content changes)
Line 26: Line 26:


* Push your patch to [[gerrit]] using [[git-review]]. You will get a '''change''' number.
* Push your patch to [[gerrit]] using [[git-review]]. You will get a '''change''' number.
 
* Choose a puppet master server <master>
* Choose a puppet master server <master>
* Build &lt;worker&gt; as a large Jessie instance (probably anywhere, but ideally in the same project as &lt;master&gt; if &lt;master&gt; is a VM
 
* On &lt;worker&gt;, apply the classes "role::puppet_compiler" and "profile::puppetmaster::labsenc" (via Horizon, for example)
* Build <worker> as a large Jessie instance (probably anywhere, but ideally in the same project as <master> if <master> is a VM
* On &lt;worker&gt;, apply the following hiera (via Horizon, for example): <syntaxhighlight lang="yaml">
 
* On <worker>, apply the classes "role::puppet_compiler" and "profile::puppetmaster::labsenc" (via Horizon, for example)
 
* On <worker>, apply the following hiera (via Horizon, for example):
<pre>
discovery::app_routes: {}
discovery::app_routes: {}
etcd::cluster_name: <something arbitrary and distinctive>
etcd::cluster_name: <something arbitrary and distinctive>
Line 44: Line 39:
etcd::use_ssl: true
etcd::use_ssl: true
labsdnsconfig: {}
labsdnsconfig: {}
</pre>
</syntaxhighlight>


* Gather facts of target host. On a local (not a VM) with keys installed to access all the necessary hosts, run this script from the puppet repo:
* Gather facts of target host. On a local (not a VM) with keys installed to access all the necessary hosts, run this script from the puppet repo:
<pre>
<syntaxhighlight lang="shell-session">
$ PUPPET_MASTERS=<master> PUPPET_COMPILER=<worker> modules/puppet_compiler/files/compiler-update-facts
$ PUPPET_MASTERS=<master> PUPPET_COMPILER=<worker> modules/puppet_compiler/files/compiler-update-facts
</pre>
</syntaxhighlight>


* In order to see results in HTML for a web browser, you will need to set up a web proxy (ideally named puppet-compiler.wmflabs.org) pointing to <worker>
* In order to see results in HTML for a web browser, you will need to set up a web proxy (ideally named puppet-compiler.wmflabs.org) pointing to &lt;worker&gt;
Make sure port 80 is open on <worker> so the proxy can reach it. You can do this via Horizon.
Make sure port 80 is open on &lt;worker&gt; so the proxy can reach it. You can do this via Horizon.


* Once all that is done, you can invoke a compiler test on <worker> like this:
* Once all that is done, you can invoke a compiler test on &lt;worker&gt; like this:
<pre>
<syntaxhighlight lang="shell-session">
CHANGE=<patch of interest> NODES=<node of interest> BUILD_NUMBER=8 puppet-compiler
CHANGE=<patch of interest> NODES=<node of interest> BUILD_NUMBER=8 puppet-compiler
</pre>
</syntaxhighlight>


The source of this information is:
The source of this information is:
* Andrew's comment in https://phabricator.wikimedia.org/T97081#3681225
* Andrew's comment in [[phab:T97081#3681225]]


== Local run (pcc utility) ==
== Local run (pcc utility) ==
Line 67: Line 62:


Example:
Example:
<pre>
<syntaxhighlight lang="shell-session">
./utils/pcc GERRIT_CHANGE_NUMBER LIST_OF_NODES --username YOUR_USERNAME --api-token 12312312312312313   
$ ./utils/pcc GERRIT_CHANGE_NUMBER LIST_OF_NODES --username YOUR_USERNAME --api-token 12312312312312313   
./utils/pcc 282936 oxygen.eqiad.wmnet --username batman --api-token 12312312312312313
$ ./utils/pcc 282936 oxygen.eqiad.wmnet --username batman --api-token 12312312312312313
</pre>
</syntaxhighlight>


== Troubleshooting ==
== Troubleshooting ==
Line 87: Line 82:
* [[Puppet migration#Puppet Catalogs compiler]]
* [[Puppet migration#Puppet Catalogs compiler]]
* [[Puppet coding/Testing#Puppet Compiler Jenkins Job]]
* [[Puppet coding/Testing#Puppet Compiler Jenkins Job]]
* Phabricator related tickets:  https://phabricator.wikimedia.org/T97081
* {{phabT|97081}}


[[Category:Puppet]]
[[Category:Puppet]]
[[Category:Documentation]]
[[Category:Documentation]]
[[Category:Cloud Services]]
[[Category:Cloud VPS]]
[[Category:Cloud VPS]]

Revision as of 04:56, 15 November 2017

You can run puppet-compiler by hand to get the results of a given puppet configuration without having to deploy it to servers.

This page gives some instructions on doing so.

Integration Jenkins

There is a jenkins job that takes a gerrit change and runs the compiler.

Steps:

  1. Push your change to gerrit using git-review
  2. Go to https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/
  3. Go to "Build with parameters" https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/build
  4. In the form, fill change number (from gerrit) and list of nodes
  5. Hit the Build button
  6. Wait for the jenkins job to end
  7. You can check for results in the jenkins Console output
  8. You can see the compiled catalogs in a web frontend, something like: https://puppet-compiler.wmflabs.org/compiler02/8755/

Local run (of jenkins job)

You can do a local run of the integration jenkins job. This is usually done in an instance VM on CloudVPS.

Steps are:

  • Push your patch to gerrit using git-review. You will get a change number.
  • Choose a puppet master server <master>
  • Build <worker> as a large Jessie instance (probably anywhere, but ideally in the same project as <master> if <master> is a VM
  • On <worker>, apply the classes "role::puppet_compiler" and "profile::puppetmaster::labsenc" (via Horizon, for example)
  • On <worker>, apply the following hiera (via Horizon, for example):
    discovery::app_routes: {}
    etcd::cluster_name: <something arbitrary and distinctive>
    etcd::cluster_state: new
    etcd::host: '%{::fqdn}'
    etcd::peers_list: <worker hostname>=http://<worker fqdn>:2380
    etcd::port: 2739
    etcd::ssl::puppet_cert_name: '%{::fqdn}'
    etcd::use_ssl: true
    labsdnsconfig: {}
    
  • Gather facts of target host. On a local (not a VM) with keys installed to access all the necessary hosts, run this script from the puppet repo:
$ PUPPET_MASTERS=<master> PUPPET_COMPILER=<worker> modules/puppet_compiler/files/compiler-update-facts
  • In order to see results in HTML for a web browser, you will need to set up a web proxy (ideally named puppet-compiler.wmflabs.org) pointing to <worker>

Make sure port 80 is open on <worker> so the proxy can reach it. You can do this via Horizon.

  • Once all that is done, you can invoke a compiler test on <worker> like this:
CHANGE=<patch of interest> NODES=<node of interest> BUILD_NUMBER=8 puppet-compiler

The source of this information is:

Local run (pcc utility)

There is a also a tool called pcc under the operations/puppet/utils repo. You'll need your Jenkins API token to make it work, retrievable under https://integration.wikimedia.org/ci/user/$YOURUSERNAME/configure.

Example:

$ ./utils/pcc GERRIT_CHANGE_NUMBER LIST_OF_NODES --username YOUR_USERNAME --api-token 12312312312312313  
$ ./utils/pcc 282936 oxygen.eqiad.wmnet --username batman --api-token 12312312312312313

Troubleshooting

Some common errors and mistakes.

  • Catalog for Cloud VPS instances doesn't get any labs classes/roles

This happens because $::realm is not set to labs. There are patches in place to fix this, but the puppet-compiler software needs to be released with these patches.

See also