You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Puppet coding/testing: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>CDanis
m (see also...)
imported>Dzahn
(Dzahn moved page Puppet coding/testing to Puppet/testing: requested by Brett )
(5 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Draft|This is a work in progress. As of March 22nd 2018.}}
#REDIRECT [[Puppet/testing]]
 
{{see also|Puppet coding/Testing}}
 
We have a set of helpers to lint, check style and even test the Puppet code we write.  This part cover how to run the utilities, how to write your own tests and even how to debug!
 
== Running tests ==
Some puppet modules have test suites using the ruby test runner rspec<ref>https://rspec.info/, Behaviour Driven Development for Ruby</ref> with rspec-puppet<ref>http://rspec-puppet.com/, RSpec test framework for your Puppet manifests</ref> and a set of rake tasks to run linting check (to validate manifests, puppet-lint, hiera yaml files, erb templates). The ruby dependencies are listed in <code>Gemfile</code> at the root of each git repositories (operations/puppet, operations/puppet/nginx ...). The dependencies are to be  installed using [https://bundler.io/ bundler] (the ruby world package manager).
 
To install all required dependencies: <source lang="bash" inline>bundle install</source> and to run a command in that environment: <source lang="bash" inline>bundle exec <some command></source>.
 
Assuming a puppet module has a <code>Rakefile</code> and tests defined in a <code>./spec</code> sub directory, one can run syntax checks, style and tests via the three commands:
 
<source lang="bash">
bundle exec rake syntax
bundle exec rake puppet-lint
bundle exec rake spec
</source>
 
== Rake explained ==
 
The <code>/Gemfile</code> asks for the ruby gem [https://rubygems.org/gems/puppetlabs_spec_helper puppetlabs_spec_helper] ([http://www.rubydoc.info/gems/puppetlabs_spec_helper/ doc]) which contains several predefined rake tasks. Hence in a module one just have to create a Rakefile with:
 
<source lang=ruby>
require 'puppetlabs_spec_helper/rake_tasks'
</source>
 
In the module, <code>rake -T</code> gives the list of all available tasks (and <code>rake -P</code> list the dependency tree), though most would do nothing:
<source lang=console>
$ cd modules/mymodule
$ bundle exec rake -T
rake beaker                # Run beaker acceptance tests
rake beaker:sets          # List available beaker nodesets
rake beaker:ssh[set,node]  # Try to use vagrant to login to the Beaker node
rake build                # Build puppet module package
rake check:dot_underscore  # Fails if any ._ files are present in directory
rake check:git_ignore      # Fails if directories contain the files specified in .gitignore
rake check:symlinks        # Fails if symlinks are present in directory
rake check:test_file      # Fails if .pp files present in tests folder
rake clean                # Clean a built module package
rake compute_dev_version  # Print development version of module
rake help                  # Display the list of available rake tasks
rake lint                  # Run puppet-lint
rake parallel_spec        # Parallel spec tests
rake release_checks        # Runs all necessary checks on a module in preparation for a release
rake rubocop              # Run RuboCop
rake rubocop:auto_correct  # Auto-correct RuboCop offenses
rake spec                  # Run spec tests and clean the fixtures directory if successful
rake spec_clean            # Clean up the fixtures directory
rake spec_prep            # Create the fixtures directory
rake spec_standalone      # Run spec tests on an existing fixtures directory
rake syntax                # Syntax check Puppet manifests and templates
rake syntax:hiera          # Syntax check Hiera config files
rake syntax:manifests      # Syntax check Puppet manifests
rake syntax:templates      # Syntax check Puppet templates
rake validate              # Check syntax of Ruby files and call :syntax and :metadata_lint
$
</source>
 
The <code>syntax*</code> tasks come from the [https://rubygems.org/gems/puppet-syntax rubygem puppet-syntax].
 
The <code>spec*</code> tasks are helpers to prepare a puppet environment to run <code>rspec</code> into:
 
* <code>spec</code> setup a test environment and run the tests
* <code>spec_prep</code> adds fixtures and module dependencies for the test environment
* <code>spec_standalone</code> run tests, assuming the test environment has been previously setup (with <code>spec_pre</code> or <code>spec</code>).
* <code>spec_clean</code> tears down that environment
 
== Writing tests ==
 
To test puppet resources, we rely on [https://github.com/rodjek/rspec-puppet rspec-puppet] an helper on top of the ruby test runner rspec. <code>rspec-puppet</code> provides utilities to setup puppet, to compile a catalog and it provides built-in assert methods to run against the generated catalog. The setup recommendation is to point puppet <code>manifest_dir</code> and <code>module_path</code> to an empty directory <code>spec/fixtures</code> that is populated automatically by the puppetlabs_spec_helper rake task <code>spec_prep</code> (which is conveniently a prerequisite of the task <code>spec</code>).
 
A minimal case requires:
* a <code>Rakefile</code>
* an helper file <code>spec/spec_helper.rb</code> which will be loaded by each test
* a spec defining the tests to conduct
 
At first the Rakefile reuses the puppetlabs_spec_helper rake tasks described in the previous section:
 
<code>Rakefile</code>:
<source lang=ruby>
require 'puppetlabs_spec_helper/rake_tasks'
</source>
 
The tests are placed in sub directories of <code>spec/</code> based on the type of Puppet resource being tested. That convention lets rspec-puppet properly setup the rspec helpers for the type of puppet resource being tested. rspec finds tests by crawling the hierachy under <code>spec</code> looking for files with the suffix <code>_spec.rb</code>. The hierarchy is:
 
<pre>
spec/
  ├── applications/
  ├── classes/
  ├──── someclass_spec.rb
  ├── defines/
  ├── functions/
  ├──── some_function_spec.rb
  ├── hosts/
  ├── types/
  └── types_aliases/
</pre>
 
We then need common code to initialize Puppet and point it to the fixture directory. That is where puppetlabs_spec_helper will create a dummy <code>manifests/site.pp</code> and eventually inject additional modules required for tests.
 
Create <code>spec/spec_helper.rb</code>:
<source lang=ruby>
require 'rspec-puppet'
 
# The empty fixture dir will be mymodule/spec/fixtures
fixture_path = File.expand_path(File.join(__FILE__, '..', 'fixtures'))
 
RSpec.configure do |c|
  # Configure puppet
  c.module_path = File.join(fixture_path, 'modules')
  c.manifest_dir = File.join(fixture_path, 'manifests')
end
</source>
 
The file will be required by each of the specs using <source  inline lang=ruby>require 'spec_helper'</source> and setup Puppet to point to <code>mymodule/spec/fixtures</code> directory (for now empty).
 
 
Given a puppet module <code>mymodule</code> consisting of a single class in <code>manifests/init.pp</code>:
<source lang=ruby>
class mymodule {
}
</source>
 
We first have to instruct puppetlabs_spec_helper to inject our module in the fixture directory. To do so create a <code>.fixtures.yml</code> at the root of the module (eg: give a puppet module mymodule:<code>/modules/mymodule/.fixtures.yml</code>.
 
<source lang=yaml>
fixtures:
    symlinks:
        mymodule: "#{source_dir}"
</source>
 
The puppet labs spec_helper task <code>spec_prep</code> would process that file and symlink our module as <code>spec/fixtures/modules/mymodule</code> as well as create an empty <code>spec/fixtures/manifests/site.pp</code>.  Late one can symlink other modules (eg: <source lang="yaml" inline>stdlib: "../../../../stdlib"</source>).
 
Since we will test a class, we create our test file under <code>spec/classes/</code> as <code>mymodule_spec.rb</code>:
<source lang=ruby>
# Helper from spec/spec_helper.rb that with the puppet configuration for rspec-puppet
require 'spec_helper'
 
# We will act on the resource "my module"
# Defined as a class resource since the file is under spec/classes
describe 'mymodule' do
  # Check whether puppet can compile the catalog for the 'mymodule' class
  it { should.compile }
end
</source>
 
Finally some fancy configuration of rspec via <code>/.rspec</code>:
<pre>
--format doc
--color
</pre>
 
And we can finally get the test environment prepared and run the spec:
<pre>
$ bundle exec rake spec
 
mymodule
  should compile into a catalogue without dependency cycles
 
Finished in 0.07349 seconds (files took 0.4312 seconds to load)
1 example, 0 failures
$
</pre>
 
Had we had an error in the manifest, for example a missing curly brace:
<pre>
$ bundle exec rake spec
 
mymodule
  should compile into a catalogue without dependency cycles (FAILED - 1)
 
Failures:
 
  1) mymodule should compile into a catalogue without dependency cycles
    Failure/Error: it { should compile }
      error during compilation:
        Syntax error at end of file; expected '}'
        at modules/mymodule/spec/fixtures/modules/mymodule/manifests/init.pp:2 on node johndoe
    # ./spec/classes/mymodule_spec.rb:4:in `block (2 levels) in <top (required)>'
 
Finished in 0.06745 seconds (files took 0.4104 seconds to load)
1 example, 1 failure
 
Failed examples:
 
rspec ./spec/classes/mymodule_spec.rb:4 # mymodule should compile into a catalogue without dependency cycles
</pre>
 
== Debugging ==
 
A collection of tips to debug spec failures.
 
=== Puppet debug log ===
Enable puppet debug log to the console. In the <code>spec_helper.rb</code> add:
<source lang=ruby>
RSpec.configure do |conf|
  if ENV['PUPPET_DEBUG']
    conf.before(:each) do
      Puppet::Util::Log.level = :debug
      Puppet::Util::Log.newdestination(:console)
    end
  end
end
</source>
Then run tests with <code>PUPPET_DEBUG=1 bundle exec rake spec</code>
 
Credits: [https://gist.github.com/maxlinc/6382696 maxlinc@github gist]).
 
=== Run a single spec / example ===
 
At first prepare the fixture environment:
<source lang="bash">
bundle exec rake spec_prep
</source>
 
Then run in the bundle environment, run rspec on a specific spec:
<source lang="bash">
bundle exec rspec spec/classes/someclass_spec.rb
</source>
 
Or you can filter based on the spec name:
<source lang="bash">
bundle exec rspec --example mymodule::someclass
</source>
 
See rspec help for more details.
 
=== Pass options to rspec from env ===
 
You can pass extra options to rspec via <code>SPEC_OPTS</code> environment variable. Useful when you want to invoke your tests from rake but want to refine what rspec does:
<source lang="bash">
SPEC_OPTS="--example mymodule::someclass" bundle exec rake
</source>
 
Which would be the equivalent of:
<source lang="bash">
bundle exec rake spec_prep
bundle exec rspec --example my module::someclass
</source>
 
=== ruby debugger ===
 
You can use the gem <code>pry</code> to break on error and get shown a console in the context of the failure.  To your Gemfile add <code>gem 'pry'</code> and install it with <code>bundle install</code> then to break inside a spec:
<source lang=ruby>
require 'spec_helper'
require 'pry'
 
describe 'mymodule::someclass' do
 
  it {
    # enable debugger
    binding.pry
    # compilation that fails:
    should compile
  }
end
</source>
 
You will then be in a console before the breakage that let you inspect the environment (<code>ls</code>). See https://github.com/pry/pry for details.
 
[https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#ruby_debug_tip reference]
 
== Integration with Jenkins ==
 
Jenkins job simply runs <code>rake test</code> ([https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#Ruby CI entry point]) from the root of the operations/puppet.git. The checks we want to run automatically are marked as prerequisites of the test task, for example:
<source lang=ruby>
task test: [:rubocop, :puppetlint_head, :syntax_head, :spec]
</source>
 
The tasks suffixed with <code>_head</code> are optimized to have the utility to only run on files changed in the proposed patch. Typically puppet-lint takes minutes to run against all the puppet manifests, when for CI we only are interested in the manifests that are actually being changed.
 
The rakefile add a task for each module having a <code>spec</code> directory. The task is named after the module and put under the namespace <code>spec</code>. Hence as soon as you create a basic structure for a module mymodule, you can run it from the root of the repository with:
<source lang="bash">
bundle exec rake spec:mymodule
</source>
And it is dynamically made a prerequisite of the <code>spec</code> task which is run by CI.  To say it otherwise, once a spec directory is created, Jenkins will try to run the spec.
 
== Resources ==
 
* http://rspec-puppet.com/ (might not be up-to-date)
* https://github.com/rodjek/rspec-puppet
* https://github.com/puppetlabs/puppetlabs_spec_helper#puppet-labs-spec-helper (among others: [https://github.com/puppetlabs/puppetlabs_spec_helper#using-fixtures doc about fixtures]).

Revision as of 18:02, 29 June 2022

Redirect to: