You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Puppet coding/testing: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Quiddity
m (fixes)
imported>Jbond
Line 2: Line 2:


We have a set of helpers to lint, check style and even test the Puppet code we write.  This part cover how to run the utilities, how to write your own tests and even how to debug!
We have a set of helpers to lint, check style and even test the Puppet code we write.  This part cover how to run the utilities, how to write your own tests and even how to debug!
=== Configuring the environment - Bundler ===
The puppet repo uses bundler to create a static environment for all dependencies required for puppet CI.  bundler looks at the Gemfile in the root directory and installs the necessary gems in the specified location.  The default location is to install in the same place as <code>gem install</code> i.e. into the system path.  this is often not desired as such its better to specify a default directory first, which causes bundler to act a bit more like virtualenv.
run the following commands to configure the local repo directory and install the various gem files locally
<syntaxhighlight lang="shell">
$ git clone ssh://gerrit.wikimedia.org:29418/operations/puppet                                                 
Cloning into 'puppet'...
remote: Counting objects: 102, done
remote: Finding sources: 100% (102/102)
remote: Getting sizes: 100% (40/40)
remote: Compressing objects: 100% (88490/88490)
remote: Total 669923 (delta 51), reused 669899 (delta 47)
Receiving objects: 100% (669923/669923), 139.63 MiB | 12.58 MiB/s, done.
Resolving deltas: 100% (531219/531219), done.
$ bundle config set --local path '.bundle/vendor'
$ cat .bundle/config                                                                                       
---
BUNDLE_PATH: ".bundle/vendor"
$ bundle install
$ bundle install                                                                                           
Fetching gem metadata from https://rubygems.org/........
Resolving dependencies.......
Fetching rake 12.0.0
Installing rake 12.0.0
---SNIP---
Fetching rbnacl 4.0.2
Installing rbnacl 4.0.2
Bundle complete! 29 Gemfile dependencies, 112 gems now installed.
Bundled gems are installed into `./.bundle/vendor`
Post-install message from minitar:
The `minitar` executable is no longer bundled with `minitar`. If you are
expecting this executable, make sure you also install `minitar-cli`.
</syntaxhighlight>


== Running tests ==
== Running tests ==
Line 15: Line 50:
bundle exec rake spec
bundle exec rake spec
</syntaxhighlight>
</syntaxhighlight>
=== Ignoring lint warnings and errors ===
If you want you can always ignore specific warnings/errors by surrounding the relevant lines with a special "lint::ignore"-comment. For example, this would ignore "WARNING: top-scope variable being used without an explicit namespace" on line 54.
<pre>
53        # lint:ignore:variable_scope
54        default    => $fqdn
55        # lint:endignore
</pre>
You can get all the names of the separate checks with '''puppet-lint --help'''. More info is on http://puppet-lint.com/controlcomments/
=== Find out which warnings/errors are currently ignored ===
There are 2 parts to this. The first is to check the global .puppet-lint.rc file in the root of the puppet repo. You will see that the following 4 checks are currently ignored globally:
*--no-80-chars-check (This is about the lines over 80 chars, we are not planning to remove this exception)
*--no-autoloader_layout-check (This is about moving all remaining class out of manifests/role/ and https://phabricator.wikimedia.org/T119042 we want this fixed)
*--no-puppet-url_without_modules-check (Yes, we want this fixed)
*--no-documentation-check (https://phabricator.wikimedia.org/T127797)
The second part is checking for individual ignores, find these with a '''grep -r "lint:ignore" *''' in the root of the puppet repo. You can help by fixing and removing any of these remaining issues.
The tracking task for getting this to perfection is https://phabricator.wikimedia.org/T93645.


== Rake explained ==
== Rake explained ==


The <code>/Gemfile</code> asks for the ruby gem [https://rubygems.org/gems/puppetlabs_spec_helper puppetlabs_spec_helper] ([http://www.rubydoc.info/gems/puppetlabs_spec_helper/ doc]) which contains several predefined rake tasks. Hence in a module one just have to create a Rakefile with:
=== Global tasks ===
The <code>Gemfile</code> we use at the repo root asks for the ruby gem [https://rubygems.org/gems/puppetlabs_spec_helper puppetlabs_spec_helper] ([http://www.rubydoc.info/gems/puppetlabs_spec_helper/ doc]) which contains several predefined rake tasks. We have also added a lot of our own tasks which are mostly defined either directly in the [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/Rakefile Rakefile] or the [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/rake_modules/taskgen.rb taskgen ruby file].  The following is a list of global tasks which may be useful to users
 
<syntaxhighlight lang="shell">
$ bundle exec rake -T                                                                                        [13:25:40]
Cloning into '/home/jbond/tmp/puppet/spec/fixtures/private'...
remote: Total 11739 (delta 0), reused 11739 (delta 0)
Receiving objects: 100% (11739/11739), 2.67 MiB | 1.41 MiB/s, done.
Resolving deltas: 100% (7210/7210), done.
rake global:doc                                                        # Build documentation
rake global:parallel_spec                                              # run Global rspec using parralel_spec (this is experiment...
rake global:spec                                                      # Run all spec tests found in modules
rake global:wmf_style                                                  # Run the wmf style guide check on all files, or on a sing...
rake help                                                              # Display the list of available rake tasks / Show the help
rake lint_fix                                                          # Run puppet-lint
rake parallel_spec                                                    # Run spec tests in parallel and clean the fixtures direct...
rake puppet_lint                                                      # Run puppet-lint
rake rubocop                                                          # Run RuboCop
rake rubocop:auto_correct                                              # Auto-correct RuboCop offenses
rake spdx:check:all                                                    # Check all files
rake spdx:check:changed                                                # Check changed files
rake spdx:check:missing_permission                                    # Check all files
rake spdx:convert:module[module]                                      # Convert a module to SPDX
rake spdx:convert:profile[profile]                                    # Convert a profile to SPDX
rake spdx:convert:role[role]                                          # Convert a profile to SPDX
rake spec                                                              # Run spec tests and clean the fixtures directory if succe...
rake strings:generate[patterns,debug,backtrace,markup,json,yard_args]  # Generate Puppet documentation with YARD
rake syntax                                                            # Syntax check Puppet manifests and templates
rake syntax:hiera                                                      # Syntax check Hiera config files
rake syntax:manifests                                                  # Syntax check Puppet manifests
rake syntax:templates                                                  # Syntax check Puppet templates
rake test                                                              # Run all actual tests in parallel for changes in HEAD
rake tox                                                              # Run all the tox-related tasks
rake tox:commit_message                                                # Check commit message
rake typos                                                            # Check common typos from /typos
rake validate                                                          # Check syntax of Ruby files and call :syntax and :metadat...
rake wmf_styleguide                                                    # Check wmf styleguide violations in the current commit
rake wmf_styleguide_delta                                              # Check regressions for the wmf style guide
</syntaxhighlight>
 
out of the above list i think the following are the most usefull to point out
* <code>bundle exec rake test</code>: this is the task used by CI.  it essentially runs all tests defined in taskgen.rb
* <code>bundle exec rake syntax</code>: run basic syntax checks, ideally your editor will catch anything mentioned here
* <code>bundle exec rake static</code>: Run WMF specific linting like tasks e.g. json schem checks
* <code>bundle exec rake test</code>: Run WMF specific unit style test e.g. tox jobs
* <code>bundle exec tox</code>: run the CI test handled by tox
* <code>bundle exec wmf_styleguide_delta</code>: run the puppet-lint tests with the wmf_styleguide checks enabled.  the delta variant ensures we only test committed/changed files
* <code>bundle exec rake rubocop:auto_correct</code> fix easily resolved ruby violations
 
We also have the ability to run rspec globally with <code>bundle exec rake global:parallel_spec</code> or <code>bundle exec rake global:spec</code> however this can take quite a bit of time and its often better to run theses test on just the files module you have changed
 
At the global level we also have the SPDX tasks, theses tasks are designed to make it easier to add SPDX headers to modules, roles and profiles in our puppet repo.  Please see the original task [https://phabricator.wikimedia.org/T308013 T308013] for more contex
 
=== Module task ===
 
For module level Rake files we often just add include the puppetlabs tasks as well as a small script to make some config changes


<syntaxhighlight lang=ruby>
<syntaxhighlight lang=ruby>
require 'puppetlabs_spec_helper/rake_tasks'
require 'puppetlabs_spec_helper/rake_tasks'
require_relative '../../rake_modules/module_rake_tasks.rb'
</syntaxhighlight>
</syntaxhighlight>


Line 65: Line 181:
* <code>spec_clean</code> tears down that environment
* <code>spec_clean</code> tears down that environment


== Writing tests ==
== Rspec ==


To test puppet resources, we rely on [https://github.com/rodjek/rspec-puppet rspec-puppet] an helper on top of the ruby test runner rspec. <code>rspec-puppet</code> provides utilities to setup puppet, to compile a catalog and it provides built-in assert methods to run against the generated catalog. The setup recommendation is to point puppet <code>manifest_dir</code> and <code>module_path</code> to an empty directory <code>spec/fixtures</code> that is populated automatically by the puppetlabs_spec_helper rake task <code>spec_prep</code> (which is conveniently a prerequisite of the task <code>spec</code>).
To test puppet resources, we rely on [https://github.com/rodjek/rspec-puppet rspec-puppet] an helper on top of the ruby test runner rspec. <code>rspec-puppet</code> provides utilities to setup puppet, to compile a catalog and it provides built-in assert methods to run against the generated catalog.  


A minimal case requires:
A minimal case requires:
* a <code>Rakefile</code>
* a <code>Rakefile</code>
* an helper file <code>spec/spec_helper.rb</code> which will be loaded by each test
* a spec defining the tests to conduct
* a spec defining the tests to conduct


Line 79: Line 194:
<syntaxhighlight lang=ruby>
<syntaxhighlight lang=ruby>
require 'puppetlabs_spec_helper/rake_tasks'
require 'puppetlabs_spec_helper/rake_tasks'
require_relative '../../rake_modules/module_rake_tasks.rb'
</syntaxhighlight>
</syntaxhighlight>


Line 96: Line 212:
</pre>
</pre>


We then need common code to initialize Puppet and point it to the fixture directory. That is where puppetlabs_spec_helper will create a dummy <code>manifests/site.pp</code> and eventually inject additional modules required for tests.


Create <code>spec/spec_helper.rb</code>:
We have created a global shared_spec file which is responsible for configurring a fixtures directory as well as configuring the private repo and injecting some default facts and global values.  in order to use this you need to add the following to any spec file you create
 
<syntaxhighlight lang=ruby>
<syntaxhighlight lang=ruby>
require 'rspec-puppet'
require_relative '../../../../rake_modules/spec_helper'
 
# The empty fixture dir will be mymodule/spec/fixtures
fixture_path = File.expand_path(File.join(__FILE__, '..', 'fixtures'))
 
RSpec.configure do |c|
  # Configure puppet
  c.module_path = File.join(fixture_path, 'modules')
  c.manifest_dir = File.join(fixture_path, 'manifests')
end
</syntaxhighlight>
</syntaxhighlight>


The file will be required by each of the specs using <syntaxhighlight  inline lang=ruby>require 'spec_helper'</syntaxhighlight> and setup Puppet to point to <code>mymodule/spec/fixtures</code> directory (for now empty).




Given a puppet module <code>mymodule</code> consisting of a single class in <code>manifests/init.pp</code>:
Given a puppet module <code>mymodule</code> consisting of a single class in <code>manifests/init.pp</code>:
<syntaxhighlight lang=ruby>
<syntaxhighlight lang=puppet>
class mymodule {
class mymodule {
}
}
</syntaxhighlight>
</syntaxhighlight>


We first have to instruct puppetlabs_spec_helper to inject our module in the fixture directory. To do so create a <code>.fixtures.yml</code> at the root of the module (eg: give a puppet module mymodule:<code>/modules/mymodule/.fixtures.yml</code>.
<syntaxhighlight lang=yaml>
fixtures:
    symlinks:
        mymodule: "#{source_dir}"
</syntaxhighlight>
The puppet labs spec_helper task <code>spec_prep</code> would process that file and symlink our module as <code>spec/fixtures/modules/mymodule</code> as well as create an empty <code>spec/fixtures/manifests/site.pp</code>.  Late one can symlink other modules (eg: <syntaxhighlight lang="yaml" inline>stdlib: "../../../../stdlib"</syntaxhighlight>).


Since we will test a class, we create our test file under <code>spec/classes/</code> as <code>mymodule_spec.rb</code>:
Since we will test a class, we create our test file under <code>spec/classes/</code> as <code>mymodule_spec.rb</code>:
<syntaxhighlight lang=ruby>
<syntaxhighlight lang=ruby>
# Helper from spec/spec_helper.rb that with the puppet configuration for rspec-puppet
require_relative '../../../../rake_modules/spec_helper'
require 'spec_helper'


# We will act on the resource "my module"
# Defined as a class resource since the file is under spec/classes
describe 'mymodule' do
describe 'mymodule' do
   # Check whether puppet can compile the catalog for the 'mymodule' class
   on_supported_os(WMFConfig.test_on).each do |os, os_facts|
  it { should.compile }
    context "on #{os}" do
      let(:facts) { os_facts }
      describe 'test compilation with default parameters' do
        it { is_expected.to compile.with_all_deps }
      end
    end
  end
end
end
</syntaxhighlight>
</syntaxhighlight>


Finally some fancy configuration of rspec via <code>/.rspec</code>:
The [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/examples/spec/classes/basic_class_spec.rb above file] along with a [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/examples/spec/classes/commented_class_spec.rb commented version] with more explanation is avalible in the puppet repo
<pre>
 
--format doc
--color
</pre>


And we can finally get the test environment prepared and run the spec:
And we can finally get the test environment prepared and run the spec:
<pre>
<syntaxhighlight lang="shell">
$ bundle exec rake spec
$ /usr/bin/ruby2.7 -I/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/lib:/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-support-3.11.0/lib /home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/exe/rspec --pattern spec/\{aliases,classes,defines,functions,hosts,integration,plans,tasks,type_aliases,types,unit\}/\*\*/\*_spec.rb
...


mymodule
Finished in 0.12021 seconds (files took 2.52 seconds to load)
  should compile into a catalogue without dependency cycles
3 examples, 0 failures
 
</syntaxhighlight>
Finished in 0.07349 seconds (files took 0.4312 seconds to load)
1 example, 0 failures
$
</pre>


Had we had an error in the manifest, for example a missing curly brace:
Had we had an error in the manifest, for example a missing curly brace:
<pre>
<syntaxhighlight lang="shell">
$ bundle exec rake spec
$ bundle exec rake spec
/usr/bin/ruby2.7 -I/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/lib:/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-support-3.11.0/lib /home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/exe/rspec --pattern spec/\{aliases,classes,defines,functions,hosts,integration,plans,tasks,type_aliases,types,unit\}/\*\*/\*_spec.rb
FFF


mymodule
Failures:
  should compile into a catalogue without dependency cycles (FAILED - 1)


Failures:
  1) mymodule on debian-9-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
    Failure/Error: it { is_expected.to compile.with_all_deps }
      error during compilation: Syntax error at end of input (file: /home/jbond/tmp/puppet/modules/mymodule/manifests/init.pp) on node storage.home.arpa
    # ./spec/classes/init_spec.rb:8:in `block (5 levels) in <top (required)>'
 
  2) mymodule on debian-10-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
    Failure/Error: it { is_expected.to compile.with_all_deps }
      error during compilation: Syntax error at end of input (file: /home/jbond/tmp/puppet/modules/mymodule/manifests/init.pp) on node storage.home.arpa
    # ./spec/classes/init_spec.rb:8:in `block (5 levels) in <top (required)>'


   1) mymodule should compile into a catalogue without dependency cycles
   3) mymodule on debian-11-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
     Failure/Error: it { should compile }
     Failure/Error: it { is_expected.to compile.with_all_deps }
       error during compilation:
       error during compilation: Syntax error at end of input (file: /home/jbond/tmp/puppet/modules/mymodule/manifests/init.pp) on node storage.home.arpa
        Syntax error at end of file; expected '}'
     # ./spec/classes/init_spec.rb:8:in `block (5 levels) in <top (required)>'
        at modules/mymodule/spec/fixtures/modules/mymodule/manifests/init.pp:2 on node johndoe
     # ./spec/classes/mymodule_spec.rb:4:in `block (2 levels) in <top (required)>'


Finished in 0.06745 seconds (files took 0.4104 seconds to load)
Finished in 0.10432 seconds (files took 2.29 seconds to load)
1 example, 1 failure
3 examples, 3 failures


Failed examples:
Failed examples:


rspec ./spec/classes/mymodule_spec.rb:4 # mymodule should compile into a catalogue without dependency cycles
rspec './spec/classes/init_spec.rb[1:1:1:1]' # mymodule on debian-9-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
</pre>
rspec './spec/classes/init_spec.rb[1:2:1:1]' # mymodule on debian-10-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
rspec './spec/classes/init_spec.rb[1:3:1:1]' # mymodule on debian-11-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
 
/usr/bin/ruby2.7 -I/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/lib:/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-support-3.11.0/lib /home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/exe/rspec --pattern spec/\{aliases,classes,defines,functions,hosts,integration,plans,tasks,type_aliases,types,unit\}/\*\*/\*_spec.rb failed
</syntaxhighlight>


== Debugging ==
== Debugging ==
Line 218: Line 323:
=== Run a single spec / example ===
=== Run a single spec / example ===


At first prepare the fixture environment:
Run in the bundle environment, run rspec on a specific spec:
<syntaxhighlight lang="bash">
bundle exec rake spec_prep
</syntaxhighlight>
 
Then run in the bundle environment, run rspec on a specific spec:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
bundle exec rspec spec/classes/someclass_spec.rb
bundle exec rspec spec/classes/someclass_spec.rb
Line 269: Line 369:


[https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#ruby_debug_tip reference]
[https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#ruby_debug_tip reference]
=== [[Portal:Wikimedia VPS|Cloud VPS]] testing ===
Nontrivial puppet changes should be applied to a Cloud VPS instance before being merged into production.  This can uncover some behaviors and code interactions that don't appear during individual file tests -- for example, puppet runs frequently fail due to duplicate resource definitions that aren't obvious to the naked eye.
To test a puppet patch:
1. Create a [[Help:Standalone puppetmaster|standalone puppetmaster]]  instance.
2. Configure that instance so that it defines the class you're working on.  You can do this either via the 'configure instance' page or by editing /var/lib/git/operations/puppet/manifests/site.pp to contain something like this:
<syntaxhighlight lang="puppet">
    node this-is-my-hostname {
        include class::I::am::working::on
    }
</syntaxhighlight>
3. Run puppet a couple of times ('$ sudo puppetd -tv') until each subsequent puppet run is clean doesn't modify anything
4. Apply your patch to /var/lib/git/operations/puppet.  Do this by cherry-picking from gerrit or by rsyncing from a local working directory.
5. Run puppet again, and note the changes that this puppet run makes.  Does puppet succeed?  Are the changes what you expected?
== catalog compilation ==
You can compile the puppet catalog and see a diff and have a preview of warnings/errors.
There are several ways of doing this (Jekins, local, etc). See main article: [[Help:Puppet-compiler | puppet-compiler]].
== Manual module testing ==
A relatively simple and crude testing way is
<pre>
$ puppet apply --noop --modulepath /path/to/modules <manifest>.pp
</pre>
Do note however that this might not work if you reference stuff outside of the module hierarchy
You can get around the missing module hierarchy problem by cloning a local copy of the puppet repo and symlinking in your new module directory.
eg.
<pre>
$ git clone --branch production https://gerrit.wikimedia.org/r/operations/puppet.git
$ cd puppet/modules
$ ln -s /path/to/mymodule .
$ puppet apply --verbose --noop --modulepath=/home/${USER}/puppet/modules /path/to/mymodule/manifest/init.pp
</pre>
Users should also check out the [[Bolt]] page which allows for using a noop run to test changes


== Integration with Jenkins ==
== Integration with Jenkins ==
Line 274: Line 424:
Jenkins job simply runs <code>rake test</code> ([https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#Ruby CI entry point]) from the root of the operations/puppet.git. The checks we want to run automatically are marked as prerequisites of the test task, for example:
Jenkins job simply runs <code>rake test</code> ([https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#Ruby CI entry point]) from the root of the operations/puppet.git. The checks we want to run automatically are marked as prerequisites of the test task, for example:
<syntaxhighlight lang=ruby>
<syntaxhighlight lang=ruby>
task test: [:rubocop, :puppetlint_head, :syntax_head, :spec]
task test: [:parallel, :wmf_styleguide_delta]
</syntaxhighlight>
</syntaxhighlight>


Line 284: Line 434:
</syntaxhighlight>
</syntaxhighlight>
And it is dynamically made a prerequisite of the <code>spec</code> task which is run by CI.  To say it otherwise, once a spec directory is created, Jenkins will try to run the spec.
And it is dynamically made a prerequisite of the <code>spec</code> task which is run by CI.  To say it otherwise, once a spec directory is created, Jenkins will try to run the spec.
=== Common errors ===
==== tab character found on line .. ====
FIX: do not use tabs, use 4-space soft tabs
Have this in your vim config (<code>.vimrc</code>):
<syntaxhighlight lang=vim>
set tabstop=4
set shiftwidth=4
set softtabstop=4
set smarttab
set expandtab
</syntaxhighlight>
or use something like this (if your local path is ./wmf/puppet/) to apply it to puppet files only.
<syntaxhighlight lang=vim>
" Wikimedia style uses 4 spaces for indentation
autocmd BufRead */wmf/puppet/**/*.pp set sw=4 ts=4 et
autocmd BufNewFile */wmf/puppet/**/*.pp set sw=4 ts=4 et
</syntaxhighlight>
open the file, :retab, :wq, done.  Make sure to review the resulting change carefully before submitting it.
Or put this in your emacs config (.emacs)
<syntaxhighlight lang=emacs>
;; Puppet config with 4 spaces
(setq puppet-indent-level 4)
(setq puppet-include-indent 4)
</syntaxhighlight>
==== double quoted string containing no variables ====
FIX: use single quotes (') for all strings unless there are variables to parse in it
==== unquoted file mode ====
FIX: always quote file modes with single quotes,like:  mode => '0750'
==== line has more than 80 characters ====
FIX: wrap your lines to be less than 80 chars, if you have to, there is \<newline>.
Vim can help when writing. Place this in your .vimrc
<pre>
set textwidth=80
</pre>
==== not in autoload module layout ====
FIX: turn your code into a puppet module ([http://docs.puppetlabs.com/puppet/2.7/reference/modules_fundamentals.html Module Fundamentals])
==== ensure found on line but it's not the first attribute ====
FIX: move your "ensure =>" to the top of the resource section. (don't forget to turn a ; into a , if it was the last attribute before)
==== unquoted resource title ====
FIX: quote all resource titles, single quotes
==== top-scope variable being used without an explicit namespace ====
FIX: use an explicit namespace in variable names ([http://docs.puppetlabs.com/guides/scope_and_puppet.html Scope and Puppet])
==== class defined inside a class ====
FIX: don't define classes inside classes
==== quoted boolean value ====
FIX: do NOT quote boolean values ( => true/ => false)
==== case statement without a default case ====
FIX: add a default case to your case statement


== Resources ==
== Resources ==

Revision as of 12:28, 6 June 2022

We have a set of helpers to lint, check style and even test the Puppet code we write. This part cover how to run the utilities, how to write your own tests and even how to debug!

Configuring the environment - Bundler

The puppet repo uses bundler to create a static environment for all dependencies required for puppet CI. bundler looks at the Gemfile in the root directory and installs the necessary gems in the specified location. The default location is to install in the same place as gem install i.e. into the system path. this is often not desired as such its better to specify a default directory first, which causes bundler to act a bit more like virtualenv.

run the following commands to configure the local repo directory and install the various gem files locally

$ git clone ssh://gerrit.wikimedia.org:29418/operations/puppet                                                  
Cloning into 'puppet'...
remote: Counting objects: 102, done
remote: Finding sources: 100% (102/102)
remote: Getting sizes: 100% (40/40)
remote: Compressing objects: 100% (88490/88490)
remote: Total 669923 (delta 51), reused 669899 (delta 47)
Receiving objects: 100% (669923/669923), 139.63 MiB | 12.58 MiB/s, done.
Resolving deltas: 100% (531219/531219), done.
$ bundle config set --local path '.bundle/vendor'
$ cat .bundle/config                                                                                         
---
BUNDLE_PATH: ".bundle/vendor"
$ bundle install
$ bundle install                                                                                             
Fetching gem metadata from https://rubygems.org/........
Resolving dependencies.......
Fetching rake 12.0.0
Installing rake 12.0.0
---SNIP---
Fetching rbnacl 4.0.2
Installing rbnacl 4.0.2
Bundle complete! 29 Gemfile dependencies, 112 gems now installed.
Bundled gems are installed into `./.bundle/vendor`
Post-install message from minitar:
The `minitar` executable is no longer bundled with `minitar`. If you are
expecting this executable, make sure you also install `minitar-cli`.

Running tests

Some puppet modules have test suites using the ruby test runner rspec[1] with rspec-puppet[2] and a set of rake tasks to run linting check (to validate manifests, puppet-lint, hiera yaml files, erb templates). The ruby dependencies are listed in Gemfile at the root of each git repositories (operations/puppet, operations/puppet/nginx ...). The dependencies are to be installed using bundler (the ruby world package manager).

To install all required dependencies: bundle install and to run a command in that environment: bundle exec <some command>.

Assuming a puppet module has a Rakefile and tests defined in a ./spec sub directory, one can run syntax checks, style and tests via the three commands:

bundle exec rake syntax
bundle exec rake puppet-lint
bundle exec rake spec

Ignoring lint warnings and errors

If you want you can always ignore specific warnings/errors by surrounding the relevant lines with a special "lint::ignore"-comment. For example, this would ignore "WARNING: top-scope variable being used without an explicit namespace" on line 54.

 53         # lint:ignore:variable_scope
 54         default     => $fqdn
 55         # lint:endignore

You can get all the names of the separate checks with puppet-lint --help. More info is on http://puppet-lint.com/controlcomments/

Find out which warnings/errors are currently ignored

There are 2 parts to this. The first is to check the global .puppet-lint.rc file in the root of the puppet repo. You will see that the following 4 checks are currently ignored globally:

The second part is checking for individual ignores, find these with a grep -r "lint:ignore" * in the root of the puppet repo. You can help by fixing and removing any of these remaining issues.

The tracking task for getting this to perfection is https://phabricator.wikimedia.org/T93645.

Rake explained

Global tasks

The Gemfile we use at the repo root asks for the ruby gem puppetlabs_spec_helper (doc) which contains several predefined rake tasks. We have also added a lot of our own tasks which are mostly defined either directly in the Rakefile or the taskgen ruby file. The following is a list of global tasks which may be useful to users

$ bundle exec rake -T                                                                                         [13:25:40]
Cloning into '/home/jbond/tmp/puppet/spec/fixtures/private'...
remote: Total 11739 (delta 0), reused 11739 (delta 0)
Receiving objects: 100% (11739/11739), 2.67 MiB | 1.41 MiB/s, done.
Resolving deltas: 100% (7210/7210), done.
rake global:doc                                                        # Build documentation
rake global:parallel_spec                                              # run Global rspec using parralel_spec (this is experiment...
rake global:spec                                                       # Run all spec tests found in modules
rake global:wmf_style                                                  # Run the wmf style guide check on all files, or on a sing...
rake help                                                              # Display the list of available rake tasks / Show the help
rake lint_fix                                                          # Run puppet-lint
rake parallel_spec                                                     # Run spec tests in parallel and clean the fixtures direct...
rake puppet_lint                                                       # Run puppet-lint
rake rubocop                                                           # Run RuboCop
rake rubocop:auto_correct                                              # Auto-correct RuboCop offenses
rake spdx:check:all                                                    # Check all files
rake spdx:check:changed                                                # Check changed files
rake spdx:check:missing_permission                                     # Check all files
rake spdx:convert:module[module]                                       # Convert a module to SPDX
rake spdx:convert:profile[profile]                                     # Convert a profile to SPDX
rake spdx:convert:role[role]                                           # Convert a profile to SPDX
rake spec                                                              # Run spec tests and clean the fixtures directory if succe...
rake strings:generate[patterns,debug,backtrace,markup,json,yard_args]  # Generate Puppet documentation with YARD
rake syntax                                                            # Syntax check Puppet manifests and templates
rake syntax:hiera                                                      # Syntax check Hiera config files
rake syntax:manifests                                                  # Syntax check Puppet manifests
rake syntax:templates                                                  # Syntax check Puppet templates
rake test                                                              # Run all actual tests in parallel for changes in HEAD
rake tox                                                               # Run all the tox-related tasks
rake tox:commit_message                                                # Check commit message
rake typos                                                             # Check common typos from /typos
rake validate                                                          # Check syntax of Ruby files and call :syntax and :metadat...
rake wmf_styleguide                                                    # Check wmf styleguide violations in the current commit
rake wmf_styleguide_delta                                              # Check regressions for the wmf style guide

out of the above list i think the following are the most usefull to point out

  • bundle exec rake test: this is the task used by CI. it essentially runs all tests defined in taskgen.rb
  • bundle exec rake syntax: run basic syntax checks, ideally your editor will catch anything mentioned here
  • bundle exec rake static: Run WMF specific linting like tasks e.g. json schem checks
  • bundle exec rake test: Run WMF specific unit style test e.g. tox jobs
  • bundle exec tox: run the CI test handled by tox
  • bundle exec wmf_styleguide_delta: run the puppet-lint tests with the wmf_styleguide checks enabled. the delta variant ensures we only test committed/changed files
  • bundle exec rake rubocop:auto_correct fix easily resolved ruby violations

We also have the ability to run rspec globally with bundle exec rake global:parallel_spec or bundle exec rake global:spec however this can take quite a bit of time and its often better to run theses test on just the files module you have changed

At the global level we also have the SPDX tasks, theses tasks are designed to make it easier to add SPDX headers to modules, roles and profiles in our puppet repo. Please see the original task T308013 for more contex

Module task

For module level Rake files we often just add include the puppetlabs tasks as well as a small script to make some config changes

require 'puppetlabs_spec_helper/rake_tasks'
require_relative '../../rake_modules/module_rake_tasks.rb'

In the module, rake -T gives the list of all available tasks (and rake -P list the dependency tree), though most would do nothing:

$ cd modules/mymodule
$ bundle exec rake -T
rake beaker                # Run beaker acceptance tests
rake beaker:sets           # List available beaker nodesets
rake beaker:ssh[set,node]  # Try to use vagrant to login to the Beaker node
rake build                 # Build puppet module package
rake check:dot_underscore  # Fails if any ._ files are present in directory
rake check:git_ignore      # Fails if directories contain the files specified in .gitignore
rake check:symlinks        # Fails if symlinks are present in directory
rake check:test_file       # Fails if .pp files present in tests folder
rake clean                 # Clean a built module package
rake compute_dev_version   # Print development version of module
rake help                  # Display the list of available rake tasks
rake lint                  # Run puppet-lint
rake parallel_spec         # Parallel spec tests
rake release_checks        # Runs all necessary checks on a module in preparation for a release
rake rubocop               # Run RuboCop
rake rubocop:auto_correct  # Auto-correct RuboCop offenses
rake spec                  # Run spec tests and clean the fixtures directory if successful
rake spec_clean            # Clean up the fixtures directory
rake spec_prep             # Create the fixtures directory
rake spec_standalone       # Run spec tests on an existing fixtures directory
rake syntax                # Syntax check Puppet manifests and templates
rake syntax:hiera          # Syntax check Hiera config files
rake syntax:manifests      # Syntax check Puppet manifests
rake syntax:templates      # Syntax check Puppet templates
rake validate              # Check syntax of Ruby files and call :syntax and :metadata_lint
$

The syntax* tasks come from the rubygem puppet-syntax.

The spec* tasks are helpers to prepare a puppet environment to run rspec into:

  • spec setup a test environment and run the tests
  • spec_prep adds fixtures and module dependencies for the test environment
  • spec_standalone run tests, assuming the test environment has been previously setup (with spec_pre or spec).
  • spec_clean tears down that environment

Rspec

To test puppet resources, we rely on rspec-puppet an helper on top of the ruby test runner rspec. rspec-puppet provides utilities to setup puppet, to compile a catalog and it provides built-in assert methods to run against the generated catalog.

A minimal case requires:

  • a Rakefile
  • a spec defining the tests to conduct

At first the Rakefile reuses the puppetlabs_spec_helper rake tasks described in the previous section:

Rakefile:

require 'puppetlabs_spec_helper/rake_tasks'
require_relative '../../rake_modules/module_rake_tasks.rb'

The tests are placed in sub directories of spec/ based on the type of Puppet resource being tested. That convention lets rspec-puppet properly setup the rspec helpers for the type of puppet resource being tested. rspec finds tests by crawling the hierachy under spec looking for files with the suffix _spec.rb. The hierarchy is:

spec/
  ├── applications/
  ├── classes/
  ├──── someclass_spec.rb
  ├── defines/
  ├── functions/
  ├──── some_function_spec.rb
  ├── hosts/
  ├── types/
  └── types_aliases/


We have created a global shared_spec file which is responsible for configurring a fixtures directory as well as configuring the private repo and injecting some default facts and global values. in order to use this you need to add the following to any spec file you create

require_relative '../../../../rake_modules/spec_helper'


Given a puppet module mymodule consisting of a single class in manifests/init.pp:

class mymodule {
}


Since we will test a class, we create our test file under spec/classes/ as mymodule_spec.rb:

require_relative '../../../../rake_modules/spec_helper'

describe 'mymodule' do
  on_supported_os(WMFConfig.test_on).each do |os, os_facts|
    context "on #{os}" do
      let(:facts) { os_facts }
      describe 'test compilation with default parameters' do
        it { is_expected.to compile.with_all_deps }
      end
    end
  end
end

The above file along with a commented version with more explanation is avalible in the puppet repo


And we can finally get the test environment prepared and run the spec:

$ /usr/bin/ruby2.7 -I/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/lib:/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-support-3.11.0/lib /home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/exe/rspec --pattern spec/\{aliases,classes,defines,functions,hosts,integration,plans,tasks,type_aliases,types,unit\}/\*\*/\*_spec.rb
...

Finished in 0.12021 seconds (files took 2.52 seconds to load)
3 examples, 0 failures

Had we had an error in the manifest, for example a missing curly brace:

$ bundle exec rake spec
/usr/bin/ruby2.7 -I/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/lib:/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-support-3.11.0/lib /home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/exe/rspec --pattern spec/\{aliases,classes,defines,functions,hosts,integration,plans,tasks,type_aliases,types,unit\}/\*\*/\*_spec.rb
FFF

Failures:

  1) mymodule on debian-9-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
     Failure/Error: it { is_expected.to compile.with_all_deps }
       error during compilation: Syntax error at end of input (file: /home/jbond/tmp/puppet/modules/mymodule/manifests/init.pp) on node storage.home.arpa
     # ./spec/classes/init_spec.rb:8:in `block (5 levels) in <top (required)>'

  2) mymodule on debian-10-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
     Failure/Error: it { is_expected.to compile.with_all_deps }
       error during compilation: Syntax error at end of input (file: /home/jbond/tmp/puppet/modules/mymodule/manifests/init.pp) on node storage.home.arpa
     # ./spec/classes/init_spec.rb:8:in `block (5 levels) in <top (required)>'

  3) mymodule on debian-11-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
     Failure/Error: it { is_expected.to compile.with_all_deps }
       error during compilation: Syntax error at end of input (file: /home/jbond/tmp/puppet/modules/mymodule/manifests/init.pp) on node storage.home.arpa
     # ./spec/classes/init_spec.rb:8:in `block (5 levels) in <top (required)>'

Finished in 0.10432 seconds (files took 2.29 seconds to load)
3 examples, 3 failures

Failed examples:

rspec './spec/classes/init_spec.rb[1:1:1:1]' # mymodule on debian-9-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
rspec './spec/classes/init_spec.rb[1:2:1:1]' # mymodule on debian-10-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
rspec './spec/classes/init_spec.rb[1:3:1:1]' # mymodule on debian-11-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles

/usr/bin/ruby2.7 -I/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/lib:/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-support-3.11.0/lib /home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/exe/rspec --pattern spec/\{aliases,classes,defines,functions,hosts,integration,plans,tasks,type_aliases,types,unit\}/\*\*/\*_spec.rb failed

Debugging

A collection of tips to debug spec failures.

Dump resources

If an example fail for now obvious reason, it is sometime helpful to dump the catalog resources just before the example. One can just print it before the failing rspec expectation:

it {
  pp catalogue.resources
  should compile
}

Puppet debug log

Enable puppet debug log to the console. In the spec_helper.rb add:

RSpec.configure do |conf|
  if ENV['PUPPET_DEBUG']
    conf.before(:each) do
      Puppet::Util::Log.level = :debug
      Puppet::Util::Log.newdestination(:console)
    end
  end
end

Then run tests with PUPPET_DEBUG=1 bundle exec rake spec

Credits: maxlinc@github gist).

Run a single spec / example

Run in the bundle environment, run rspec on a specific spec:

bundle exec rspec spec/classes/someclass_spec.rb

Or you can filter based on the spec name:

bundle exec rspec --example mymodule::someclass

See rspec help for more details.

Pass options to rspec from env

You can pass extra options to rspec via SPEC_OPTS environment variable. Useful when you want to invoke your tests from rake but want to refine what rspec does:

SPEC_OPTS="--example mymodule::someclass" bundle exec rake

Which would be the equivalent of:

bundle exec rake spec_prep
bundle exec rspec --example my module::someclass

ruby debugger

You can use the gem pry to break on error and get shown a console in the context of the failure. To your Gemfile add gem 'pry' and install it with bundle install then to break inside a spec:

require 'spec_helper'
require 'pry'

describe 'mymodule::someclass' do

  it {
     # enable debugger
     binding.pry
     # compilation that fails: 
     should compile
  }
end

You will then be in a console before the breakage that let you inspect the environment (ls) or print the compiled catalogue (p catalogue). See https://github.com/pry/pry for details.

reference


Cloud VPS testing

Nontrivial puppet changes should be applied to a Cloud VPS instance before being merged into production. This can uncover some behaviors and code interactions that don't appear during individual file tests -- for example, puppet runs frequently fail due to duplicate resource definitions that aren't obvious to the naked eye.

To test a puppet patch:

1. Create a standalone puppetmaster instance.

2. Configure that instance so that it defines the class you're working on. You can do this either via the 'configure instance' page or by editing /var/lib/git/operations/puppet/manifests/site.pp to contain something like this:

    node this-is-my-hostname {
        include class::I::am::working::on
    }

3. Run puppet a couple of times ('$ sudo puppetd -tv') until each subsequent puppet run is clean doesn't modify anything

4. Apply your patch to /var/lib/git/operations/puppet. Do this by cherry-picking from gerrit or by rsyncing from a local working directory.

5. Run puppet again, and note the changes that this puppet run makes. Does puppet succeed? Are the changes what you expected?


catalog compilation

You can compile the puppet catalog and see a diff and have a preview of warnings/errors.

There are several ways of doing this (Jekins, local, etc). See main article: puppet-compiler.

Manual module testing

A relatively simple and crude testing way is

$ puppet apply --noop --modulepath /path/to/modules <manifest>.pp

Do note however that this might not work if you reference stuff outside of the module hierarchy

You can get around the missing module hierarchy problem by cloning a local copy of the puppet repo and symlinking in your new module directory.

eg.

$ git clone --branch production https://gerrit.wikimedia.org/r/operations/puppet.git
$ cd puppet/modules
$ ln -s /path/to/mymodule .
$ puppet apply --verbose --noop --modulepath=/home/${USER}/puppet/modules /path/to/mymodule/manifest/init.pp

Users should also check out the Bolt page which allows for using a noop run to test changes

Integration with Jenkins

Jenkins job simply runs rake test (CI entry point) from the root of the operations/puppet.git. The checks we want to run automatically are marked as prerequisites of the test task, for example:

task test: [:parallel, :wmf_styleguide_delta]

The tasks suffixed with _head are optimized to have the utility to only run on files changed in the proposed patch. Typically puppet-lint takes minutes to run against all the puppet manifests, when for CI we only are interested in the manifests that are actually being changed.

The rakefile add a task for each module having a spec directory. The task is named after the module and put under the namespace spec. Hence as soon as you create a basic structure for a module mymodule, you can run it from the root of the repository with:

bundle exec rake spec:mymodule

And it is dynamically made a prerequisite of the spec task which is run by CI. To say it otherwise, once a spec directory is created, Jenkins will try to run the spec.

Common errors

tab character found on line ..

FIX: do not use tabs, use 4-space soft tabs

Have this in your vim config (.vimrc):

set tabstop=4
set shiftwidth=4
set softtabstop=4
set smarttab
set expandtab

or use something like this (if your local path is ./wmf/puppet/) to apply it to puppet files only.

" Wikimedia style uses 4 spaces for indentation
autocmd BufRead */wmf/puppet/**/*.pp set sw=4 ts=4 et
autocmd BufNewFile */wmf/puppet/**/*.pp set sw=4 ts=4 et

open the file, :retab, :wq, done. Make sure to review the resulting change carefully before submitting it.

Or put this in your emacs config (.emacs)

;; Puppet config with 4 spaces
(setq puppet-indent-level 4)
(setq puppet-include-indent 4)

double quoted string containing no variables

FIX: use single quotes (') for all strings unless there are variables to parse in it

unquoted file mode

FIX: always quote file modes with single quotes,like: mode => '0750'

line has more than 80 characters

FIX: wrap your lines to be less than 80 chars, if you have to, there is \<newline>. Vim can help when writing. Place this in your .vimrc

set textwidth=80

not in autoload module layout

FIX: turn your code into a puppet module (Module Fundamentals)

ensure found on line but it's not the first attribute

FIX: move your "ensure =>" to the top of the resource section. (don't forget to turn a ; into a , if it was the last attribute before)

unquoted resource title

FIX: quote all resource titles, single quotes

top-scope variable being used without an explicit namespace

FIX: use an explicit namespace in variable names (Scope and Puppet)

class defined inside a class

FIX: don't define classes inside classes

quoted boolean value

FIX: do NOT quote boolean values ( => true/ => false)

case statement without a default case

FIX: add a default case to your case statement

Resources

  1. https://rspec.info/, Behaviour Driven Development for Ruby
  2. http://rspec-puppet.com/, RSpec test framework for your Puppet manifests