You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Bolt: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>JHathaway
(Created page with "= Puppet Bolt = Puppet Bolt supports a variety of uses cases, but one in particular is interesting for our use, masterless Puppet catalog applies. In a masterless apply rather than obtaining the catalog from the puppet master the catalog is compiled directly from the source repository. One advantage of a masterless apply is that it can offer faster feedback on code changes, since you don’t need to commit in order to apply or noop your changes against a destination hos...")
 
imported>JHathaway
(update page to point to utilities in the repo)
 
(6 intermediate revisions by 4 users not shown)
Line 1: Line 1:
= Puppet Bolt =
'''Puppet Bolt''' supports a variety of uses cases, but one in particular is interesting for our use, masterless Puppet catalog applies. In a masterless apply rather than obtaining the catalog from the puppet master the catalog is compiled directly from the source repository. One advantage of a masterless apply is that it can offer faster feedback on code changes, since you don’t need to commit in order to apply or noop your changes against a destination host, i.e.:
 
Puppet Bolt supports a variety of uses cases, but one in particular is interesting for our use, masterless Puppet catalog applies. In a masterless apply rather than obtaining the catalog from the puppet master the catalog is compiled directly from the source repository. One advantage of a masterless apply is that it can offer faster feedback on code changes, since you don’t need to commit in order to apply or noop your changes against a destination host, i.e.:


<pre># Edit
<pre># Edit
Line 23: Line 21:
$ sudo apt-get update
$ sudo apt-get update
$ sudo apt-get install puppet-bolt</pre>
$ sudo apt-get install puppet-bolt</pre>
=== Patch Bolt to read from named pipes ===
[https://github.com/puppetlabs/bolt/pull/3057 Upstream pull request]
<pre>$ sudo patch -p1 -d /opt/puppetlabs/bolt/lib/ruby/gems/2.7.0/gems/bolt-3.21.0/ &lt;&lt;EOF
commit 5d70449800c86d76f2f6775710cec39984984af5
Author: Jesse Hathaway &lt;jesse@mbuki-mvuki.org&gt;
Date:  Tue Feb 15 15:00:25 2022 -0600
    bolt apply: allow reading manifest from a named pipe
   
    Prior to this commit bolt threw an error when given a manifest which
    resolves to a named pipe:
   
        $ bolt apply -t butter.example.com &lt;(cat manifests/*.pp)
   
    After this commit bolt allows reading from the named pipes!
diff --git a/lib/bolt/util.rb b/lib/bolt/util.rb
index 2980ea3d..a28c5d14 100644
--- a/lib/bolt/util.rb
+++ b/lib/bolt/util.rb
@@ -344,7 +344,7 @@ module Bolt
        if !stat.readable?
          raise Bolt::FileError.new(&quot;The #{type} '#{path}' is unreadable&quot;, path)
-        elsif !stat.file? &amp;&amp; (!allow_dir || !stat.directory?)
+        elsif !allow_dir &amp;&amp; stat.directory?
          expected = allow_dir ? 'file or directory' : 'file'
          raise Bolt::FileError.new(&quot;The #{type} '#{path}' is not a #{expected}&quot;, path)
        elsif stat.directory?
EOF</pre>
=== Delete Bolt’s system module ===
=== Delete Bolt’s system module ===


This module conflicts with our system module, [https://github.com/puppetlabs/bolt/issues/3055 Upstream pull request] or [https://gerrit.wikimedia.org/r/c/operations/puppet/+/764884 rename our module]
This module conflicts with our system module, [https://github.com/puppetlabs/bolt/issues/3055 Upstream pull request] or [https://gerrit.wikimedia.org/r/c/operations/puppet/+/764884 rename our module]


<pre>$ sudo rm -r /opt/puppetlabs/bolt/lib/ruby/gems/2.7.0/gems/bolt-3.21.0/bolt-modules/system</pre>
<pre>$ sudo find /opt/puppetlabs/bolt -depth -path '*bolt-modules/system' -exec rm -r '{}' \+</pre>
=== Add Bolt config, labs-private, and supporting modules ===


<pre>$ cat &gt;boltup &lt;&lt;EOM
=== Add bolt utilities to your path ===
#!/bin/bash


set -o errexit
<pre>$ git clone gitlab.wikimedia.org:jhathaway/bolt-wmf.git
set -o nounset
$ cd bolt-wmf
$ cp bolt-wmf puppet-out-grep ~/.local/bin/</pre>


cat &gt;bolt-project.yaml &lt;&lt;EOF
== Usage ==
---
name: wmf
apply-settings:
  show_diff: true
hiera-config: 'hiera.yaml'
modulepath:
  - 'private/modules'
  - 'modules'
  - 'bolt/modules'
EOF


cat &gt;inventory.yaml &lt;&lt;EOF
=== Noop a server ===
config:
Saving its output into puppet.out, so we can go through it more carefully<pre>$ bolt-wmf -t mirror1001.wikimedia.org -- apply --noop | tee puppet.out</pre>
  transport: ssh
  ssh:
      interpreters:
          # Use our system ruby
          rb: /usr/bin/ruby
      # Switch to root before running puppet
      run-as: root
features:
    # Do not try to install the puppet-agent
    - puppet-agent
EOF


# setup hiera
=== Look through output ===
sed -E 's#/etc/puppet/##' modules/puppetmaster/files/production.hiera.yaml &gt;hiera.yaml


# clone repos
Use the puppet-out-grep script to remove items from the console output that you don't care about
if [[ ! -e 'private' ]]; then
    git clone git@github.com:wikimedia/labs-private.git private
fi
mkdir -p bolt/modules
if [[ ! -e 'bolt/modules/nagios_core' ]]; then
    git clone git@github.com:puppetlabs/puppetlabs-nagios_core.git bolt/modules/nagios_core
fi
if [[ ! -e 'bolt/modules/mailalias_core' ]]; then
    git clone git@github.com:puppetlabs/puppetlabs-mailalias_core.git bolt/modules/mailalias_core
fi
EOM
$ bash boltup</pre>
== Usage ==


=== Noop a server ===
<pre>$ puppet-out-grep Exim Nagios</pre>


<pre>$ bolt apply --noop -t mirror1001.wikimedia.org &lt;(cat manifests/*.pp)</pre>
== Outstanding Issues ==
== Outstanding Issues ==


# Bolt's system modules conflicts with our system module, [https://github.com/puppetlabs/bolt/issues/3055 Upstream pull request] or [https://gerrit.wikimedia.org/r/c/operations/puppet/+/764884 rename our module]
# Bolt does not preserve symlinks on files with recurse and source directories, https://github.com/puppetlabs/bolt/issues/3056
# Bolt does not preserve symlinks on files with recurse and source directories, https://github.com/puppetlabs/bolt/issues/3056
# PuppetDB queries do not work, nor do exported resources
# PuppetDB queries do not work, nor do exported resources
# <code>acme_chief</code> module does not work
# <code>acme_chief</code> module does not work
# Nooping a server with Bolt will take out a Puppet lock, which would block a cron based Puppet run which began after the Bolt run
# Nooping a server with Bolt will take out a Puppet lock, which would block a cron based Puppet run which began after the Bolt run
[[Category:Puppet]]
[[Category:SRE Infrastructure Foundations]]

Latest revision as of 19:39, 30 August 2022

Puppet Bolt supports a variety of uses cases, but one in particular is interesting for our use, masterless Puppet catalog applies. In a masterless apply rather than obtaining the catalog from the puppet master the catalog is compiled directly from the source repository. One advantage of a masterless apply is that it can offer faster feedback on code changes, since you don’t need to commit in order to apply or noop your changes against a destination host, i.e.:

# Edit
$ vi modules/example/manifests/init.pp

# Noop to see changes
$ bolt apply --noop -t node.example.com

# Commit
$ git commit -a -m 'add example module'

Other projects which support a masterless workflow:

Install Directions

Install Bolt, not currently in Debian unfortunately

$ wget https://apt.puppet.com/puppet-tools-release-bullseye.deb
$ sudo dpkg -i puppet-tools-release-bullseye.deb
$ sudo apt-get update
$ sudo apt-get install puppet-bolt

Delete Bolt’s system module

This module conflicts with our system module, Upstream pull request or rename our module

$ sudo find /opt/puppetlabs/bolt -depth -path '*bolt-modules/system' -exec rm -r '{}' \+

Add bolt utilities to your path

$ git clone gitlab.wikimedia.org:jhathaway/bolt-wmf.git
$ cd bolt-wmf
$ cp bolt-wmf puppet-out-grep ~/.local/bin/

Usage

Noop a server

Saving its output into puppet.out, so we can go through it more carefully

$ bolt-wmf -t mirror1001.wikimedia.org -- apply --noop | tee puppet.out

Look through output

Use the puppet-out-grep script to remove items from the console output that you don't care about

$ puppet-out-grep Exim Nagios

Outstanding Issues

  1. Bolt's system modules conflicts with our system module, Upstream pull request or rename our module
  2. Bolt does not preserve symlinks on files with recurse and source directories, https://github.com/puppetlabs/bolt/issues/3056
  3. PuppetDB queries do not work, nor do exported resources
  4. acme_chief module does not work
  5. Nooping a server with Bolt will take out a Puppet lock, which would block a cron based Puppet run which began after the Bolt run