You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Puppet coding/testing: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Hashar
No edit summary
 
imported>Hashar
(link to the Gerrit patch https://gerrit.wikimedia.org/r/#/c/307223/)
Line 1: Line 1:
{{Draft}}
{{Draft|This is a work in progress. As of January 25th you will need the puppet patch https://gerrit.wikimedia.org/r/#/c/307223/ to experiment running puppet tests.}}


==== Running ====
We have a set of helpers to lint, check style and even test the Puppet code we write. This part cover how to run checks, how to write your own and debug your tests!
Some modules have test suites using the ruby test runner rspec and a set of rake tasks to run linting check (valid manifests, puppet-lint, hiera yaml files, erb templates). The ruby dependencies are listed in /Gemfile and are installed via bundler. To install all dependencies and later run a command in that environment:


<code>
==== Running tests ====
Some modules have test suites using the ruby test runner rspec and a set of rake tasks to run linting check (to validate manifests, puppet-lint, hiera yaml files, erb templates). The ruby dependencies are listed in <code>Gemfile</code> at the root of the repository and are installed via [https://bundler.io/ bundler]. To install all dependencies and later run a command in that environment:
 
<source>
bundle install
bundle install
bundle exec <some command>
bundle exec <some command>
</code>
</source>


Assuming a module has a Rakefile and tests defined in a spec sub directory, one can run all tests via:
Assuming a module has a Rakefile and tests defined in a <code>spec</code> sub directory, one can run syntax checks, style and tests via the three commands:


<code>
<source>
bundle exec rake syntax
bundle exec rake syntax
bundle exec rake puppet-lint
bundle exec rake puppet-lint
bundle exec rake spec
bundle exec rake spec
</code>
</source>


You can list the various rake tasks via <code>bundle exec rake -T</code> or for a tree of the task dependencies <code>bundle exec rake -P</code>.
You can list the various rake tasks via <code>bundle exec rake -T</code> or to get a tree of the task dependencies <code>bundle exec rake -P</code>.


==== Rake explained ====
==== Rake explained ====


The <code>/Gemfile</code> ask for the rubygem [https://rubygems.org/gems/puppetlabs_spec_helper puppetlabs_spec_helper] ([www.rubydoc.info/gems/puppetlabs_spec_helper/ doc]) which contains several predefined rake tasks. Hence in a module one just have to create a Rakefile with:
The <code>/Gemfile</code> asks for the rubygem [https://rubygems.org/gems/puppetlabs_spec_helper puppetlabs_spec_helper] ([www.rubydoc.info/gems/puppetlabs_spec_helper/ doc]) which contains several predefined rake tasks. Hence in a module one just have to create a Rakefile with:


<source lang=ruby>
<source lang=ruby>
Line 58: Line 59:
The <code>syntax*</code> tasks come from the [https://rubygems.org/gems/puppet-syntax rubygem puppet-syntax].
The <code>syntax*</code> tasks come from the [https://rubygems.org/gems/puppet-syntax rubygem puppet-syntax].


The <code>spec*</code> tasks are helper to run rspec taking care of preparing test fixtures and tearing them down.
The <code>spec*</code> tasks are helpers to prepare a puppet environment to run rspec into. Notably adding fixtures and module dependencies for the test environment and tearing down the environment on test completions.


==== rspec-puppet ====
==== Writing tests ====


To tests puppet resources, we rely on [https://github.com/rodjek/rspec-puppet rspec-puppet] an helper on top of the ruby test runner rspec. rspec-puppet provides utilities to setup puppet, compile a catalog and has built-in assert methods to run against the catalog. The recommendation is to point puppet <code>manifest_dir</code> and <code>module_path</code> to an empty fixture directory that is populated via puppetlabs_spec_helper tasks <code>spec_prep</code> (a prerequisites of the task <code>spec</code>).


To tests puppet resources, we rely on [https://github.com/rodjek/rspec-puppet rspec-puppet] an helper on top of the ruby test runner rspec. rspec-puppet provides utilities to setup puppet, compile a catalog and has built-in assert methods to run against the catalog.
A minimal case requires a Rakefile, an helper file to be read by each test, and a spec. At first the Rakefile reuses the puppetlabs_spec_helper rake tasks described in the previous section:
 
A minimal case requires a Rakefile, an helper file to be read by each test, and a spec. At first the Rakefile reuse the puppetlabs_spec_helper rake tasks described in the previous section:


<code>Rakefile</code>:
<code>Rakefile</code>:
Line 72: Line 72:
</source>
</source>


The tests are placed in sub directories of <code>spec/</code> based on the type of Puppet resource being tested. That convention lets rspec-puppet properly setup the environment. rspec find tests by crawling the hierachy looking for files with the suffix <code>_spec.rb</code>. The hierarchy is:
The tests are placed in sub directories of <code>spec/</code> based on the type of Puppet resource being tested. That convention lets rspec-puppet properly setup the rspec helpers for the type of puppet resource being tested. rspec finds tests by crawling the hierachy under <code>spec</code> looking for files with the suffix <code>_spec.rb</code>. The hierarchy is:


<pre>
<pre>
Line 78: Line 78:
   ├── applications/
   ├── applications/
   ├── classes/
   ├── classes/
  ├──── someclass_spec.rb
   ├── defines/
   ├── defines/
   ├── functions/
   ├── functions/
  ├──── afunc_spec.rb
   ├── hosts/
   ├── hosts/
   ├── types/
   ├── types/
Line 85: Line 87:
</pre>
</pre>


We also need common code to initialize Puppet and point it to a fixture directory, that is where rspec-puppet will create a dummy manifests/site.pp and inject required modules.
We also need common code to initialize Puppet and point it to a fixture directory, that is where puppetlabs_spec_helper will create a dummy <code>manifests/site.pp</code> and eventually inject additional modules required for tests:


Create <code>spec/spec_helper.rb</code>:
Create <code>spec/spec_helper.rb</code>:
Line 99: Line 101:
</source>
</source>


The file will be required by each of the specs.
The file will be required by each of the specs using <source  inline lang=ruby>require 'spec_helper'</source>.




Line 108: Line 110:
</source>
</source>


We first have to instruct rspec-puppet to inject our module in the fixture directory. To do so create a <code>/.fixtures.yml</code>:
We first have to instruct rspec-puppet to inject our module in the fixture directory. To do so create a <code>.fixtures.yml</code> at the root of the module (eg: <code>/modules/foobar/.fixtures.yml</code>.
 
<source lang=yaml>
<source lang=yaml>
fixtures:
fixtures:
Line 115: Line 118:
</source>
</source>


Since we will test a class, we create a <code>mymodule_spec.rb</code> file under <code>spec/classes/</code>:
The puppet labs spec_helper task <code>spec_prep</code> would process that file and symlink our module as <code>spec/fixtures/modules/mymodule</code> as well as create an empty <code>spec/fixtures/manifests/site.pp</code>.
 
Since we will test a class, we create our test file under under <code>spec/classes/</code> as <code>mymodule_spec.rb</code>:
<source lang=ruby>
<source lang=ruby>
# Helper from spec/spec_helper.rb that with the puppet configuration for rspec-puppet
require 'spec_helper'
require 'spec_helper'


# We will act on the resource "mymodule"
# We will act on the resource "my module"
# Defined as a class resource since the file is under spec/classes
describe 'mymodule' do
describe 'mymodule' do
  # Check whether puppet can compile the catalog for the 'mymodule' class
   it { should.compile }
   it { should.compile }
end
end
Line 131: Line 139:
</pre>
</pre>


And we can run the spec:
And we can finally run get the test environment prepared and run the spec:
<pre>
<pre>
$ bundle exec rake spec
$ bundle exec rake spec
Line 143: Line 151:
</pre>
</pre>


Success!
Had we had an error in the manifest, for example a missing curly brace:
<pre>
$ bundle exec rake spec
 
mymodule
  should compile into a catalogue without dependency cycles (FAILED - 1)
 
Failures:
 
  1) mymodule should compile into a catalogue without dependency cycles
    Failure/Error: it { should compile }
      error during compilation:
        Syntax error at end of file; expected '}'
        at modules/mymodule/spec/fixtures/modules/mymodule/manifests/init.pp:2 on node johndoe
    # ./spec/classes/mymodule_spec.rb:4:in `block (2 levels) in <top (required)>'
 
Finished in 0.06745 seconds (files took 0.4104 seconds to load)
1 example, 1 failure
 
Failed examples:
 
rspec ./spec/classes/mymodule_spec.rb:4 # mymodule should compile into a catalogue without dependency cycles
</pre>
 
==== Debugging ====
 
A collection of tips to debug spec failures.
 
===== Puppet debug log =====
Enable puppet debug log to the console. In the <code>spec_helper.rb</code> add:
<source lang=ruby>
if ENV['PUPPET_DEBUG']
  Puppet::Util::Log.level = :debug
  Puppet::Util::Log.newdestination(:console)
</source>
 
Then run tests with <code>PUPPET_DEBUG=1 bundle exec rake spec</code>
 
Credits: [https://gist.github.com/maxlinc/6382696 maxlinc@github gist]).
 
===== Run a single spec / example =====
 
At first prepare the fixture environment:
<source>
bundle exec rake spec_prep
</source>
 
Then run in the bundle environment, run rspec on a specific spec:
<source>
bundle exec rspec spec/classes/someclass_spec.rb
</source>
 
Or you can filter based on the spec name:
<source>
bundle exec rspec --example mymodule::someclass
</source>
 
See rspec help for more details.
 
===== ruby debugger =====
 
 
You can use the gem <code>pry</code> to break on error and get shown a console in the context of the failure.  To your Gemfile add <code>gem 'pry'</code> and install it with <code>bundle install</code> then to break inside a spec:
<source lang=ruby>
require 'spec_helper'
require 'pry'
 
describe 'mymodule::someclass' do
 
  it {
    # enable debugger
    binding.pry
    # compilation that fails:
    should compile
  }
end
</source>
 
You will then be in a console before the breakage that let you inspect the environment (<code>ls</code>). See https://github.com/pry/pry for details.
 
[https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#ruby_debug_tip reference]
 
==== Integration with Jenkins ====
 
''Note: as of January 25th patches have not been merged. See [https://gerrit.wikimedia.org/r/#/q/topic:rspec-puppet+project:operations/puppet+NOT+is:abandoned Gerrit topic:rspec-puppet].''
 
Jenkins job simply runs <code>rake test</code> ([https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#Ruby CI entry point]) from the root of the operations/puppet.git. The checks we want to run automatically are marked as prerequisites of the test task, for example:
<source lang=ruby>
task test: [:rubocop, :puppetlint_head, :syntax_head, :spec]
</source>
 
The tasks suffixed with <code>_head</code> are optimized to have the utility to only run on files changed in the proposed patch. Typically puppet-lint takes minutes to run against all the puppet manifests, when for CI we only are interested in the manifests that are actually being changed.
 
The rakefile add a task for each module having a <code>spec</code> directory. The task is named after the module and put under the namespace <code>spec</code>. Hence as soon as you create a basic structure for a module mymodule, you can run it from the root of the repository with:
<source>
bundle exec rake spec:mymodule
</source>
And it is dynamically made a prerequisite of the <code>spec</code> task which is run by CI.  To say it otherwise, once a spec directory is created, Jenkins will try to run the spec.
 
As of January 25, patches are pending consensus.  For the CI integration, at first the job will probably only be run on demand (by commenting <code>check experimental</code> in Gerrit).  Notably the puppet catalog compilation is a bit slow (1.2sec per class) and we should run them in parallel.


==== Resources ====
==== Resources ====


https://github.com/rodjek/rspec-puppet
* http://rspec-puppet.com/ (might not be up-to-date)
* https://github.com/rodjek/rspec-puppet
* https://github.com/puppetlabs/puppetlabs_spec_helper#puppet-labs-spec-helper (among others: [https://github.com/puppetlabs/puppetlabs_spec_helper#using-fixtures doc about fixtures]).

Revision as of 15:44, 25 January 2017

We have a set of helpers to lint, check style and even test the Puppet code we write. This part cover how to run checks, how to write your own and debug your tests!

Running tests

Some modules have test suites using the ruby test runner rspec and a set of rake tasks to run linting check (to validate manifests, puppet-lint, hiera yaml files, erb templates). The ruby dependencies are listed in Gemfile at the root of the repository and are installed via bundler. To install all dependencies and later run a command in that environment:

bundle install
bundle exec <some command>

Assuming a module has a Rakefile and tests defined in a spec sub directory, one can run syntax checks, style and tests via the three commands:

bundle exec rake syntax
bundle exec rake puppet-lint
bundle exec rake spec

You can list the various rake tasks via bundle exec rake -T or to get a tree of the task dependencies bundle exec rake -P.

Rake explained

The /Gemfile asks for the rubygem puppetlabs_spec_helper ([www.rubydoc.info/gems/puppetlabs_spec_helper/ doc]) which contains several predefined rake tasks. Hence in a module one just have to create a Rakefile with:

require 'puppetlabs_spec_helper/rake_tasks'
$ cd modules/mymodule
$ bundle exec rake -T
rake beaker                # Run beaker acceptance tests
rake beaker_nodes          # List available beaker nodesets
rake build                 # Build puppet module package
rake check:dot_underscore  # Fails if any ._ files are present in directory
rake check:git_ignore      # Fails if directories contain the files specified in .gitignore
rake check:symlinks        # Fails if symlinks are present in directory
rake check:test_file       # Fails if .pp files present in tests folder
rake clean                 # Clean a built module package
rake compute_dev_version   # Print development version of module
rake coverage              # Generate code coverage information
rake help                  # Display the list of available rake tasks
rake lint                  # Run puppet-lint
rake release_checks        # Runs all nessesary checks on a module in preparation for a release
rake spec                  # Run spec tests in a clean fixtures directory
rake spec_clean            # Clean up the fixtures directory
rake spec_prep             # Create the fixtures directory
rake spec_standalone       # Run spec tests on an existing fixtures directory
rake syntax                # Syntax check Puppet manifests and templates
rake syntax:hiera          # Syntax check Hiera config files
rake syntax:manifests      # Syntax check Puppet manifests
rake syntax:templates      # Syntax check Puppet templates
rake validate              # Check syntax of Ruby files and call :syntax and :metadata_lint
$

The syntax* tasks come from the rubygem puppet-syntax.

The spec* tasks are helpers to prepare a puppet environment to run rspec into. Notably adding fixtures and module dependencies for the test environment and tearing down the environment on test completions.

Writing tests

To tests puppet resources, we rely on rspec-puppet an helper on top of the ruby test runner rspec. rspec-puppet provides utilities to setup puppet, compile a catalog and has built-in assert methods to run against the catalog. The recommendation is to point puppet manifest_dir and module_path to an empty fixture directory that is populated via puppetlabs_spec_helper tasks spec_prep (a prerequisites of the task spec).

A minimal case requires a Rakefile, an helper file to be read by each test, and a spec. At first the Rakefile reuses the puppetlabs_spec_helper rake tasks described in the previous section:

Rakefile:

require 'puppetlabs_spec_helper/rake_tasks'

The tests are placed in sub directories of spec/ based on the type of Puppet resource being tested. That convention lets rspec-puppet properly setup the rspec helpers for the type of puppet resource being tested. rspec finds tests by crawling the hierachy under spec looking for files with the suffix _spec.rb. The hierarchy is:

spec/
  ├── applications/
  ├── classes/
  ├──── someclass_spec.rb
  ├── defines/
  ├── functions/
  ├──── afunc_spec.rb
  ├── hosts/
  ├── types/
  └── types_aliases/

We also need common code to initialize Puppet and point it to a fixture directory, that is where puppetlabs_spec_helper will create a dummy manifests/site.pp and eventually inject additional modules required for tests:

Create spec/spec_helper.rb:

require 'rspec-puppet'

fixture_path = File.expand_path(File.join(__FILE__, '..', 'fixtures'))

RSpec.configure do |c|
  c.module_path = File.join(fixture_path, 'modules')
  c.manifest_dir = File.join(fixture_path, 'manifests')
end

The file will be required by each of the specs using require 'spec_helper'.


Given a puppet module mymodule consisting of a single class in manifests/init.pp:

class mymodule {
}

We first have to instruct rspec-puppet to inject our module in the fixture directory. To do so create a .fixtures.yml at the root of the module (eg: /modules/foobar/.fixtures.yml.

fixtures:
    symlinks:
        mymodule: "#{source_dir}"

The puppet labs spec_helper task spec_prep would process that file and symlink our module as spec/fixtures/modules/mymodule as well as create an empty spec/fixtures/manifests/site.pp.

Since we will test a class, we create our test file under under spec/classes/ as mymodule_spec.rb:

# Helper from spec/spec_helper.rb that with the puppet configuration for rspec-puppet
require 'spec_helper'

# We will act on the resource "my module"
# Defined as a class resource since the file is under spec/classes
describe 'mymodule' do
  # Check whether puppet can compile the catalog for the 'mymodule' class
  it { should.compile }
end

Finally some fancy configuration of rspec via /.rspec:

--format doc
--color

And we can finally run get the test environment prepared and run the spec:

$ bundle exec rake spec

mymodule
  should compile into a catalogue without dependency cycles

Finished in 0.07349 seconds (files took 0.4312 seconds to load)
1 example, 0 failures
$

Had we had an error in the manifest, for example a missing curly brace:

$ bundle exec rake spec

mymodule
  should compile into a catalogue without dependency cycles (FAILED - 1)

Failures:

  1) mymodule should compile into a catalogue without dependency cycles
     Failure/Error: it { should compile }
       error during compilation:
         Syntax error at end of file; expected '}'
         at modules/mymodule/spec/fixtures/modules/mymodule/manifests/init.pp:2 on node johndoe
     # ./spec/classes/mymodule_spec.rb:4:in `block (2 levels) in <top (required)>'

Finished in 0.06745 seconds (files took 0.4104 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/classes/mymodule_spec.rb:4 # mymodule should compile into a catalogue without dependency cycles

Debugging

A collection of tips to debug spec failures.

Puppet debug log

Enable puppet debug log to the console. In the spec_helper.rb add:

if ENV['PUPPET_DEBUG']
  Puppet::Util::Log.level = :debug
  Puppet::Util::Log.newdestination(:console)

Then run tests with PUPPET_DEBUG=1 bundle exec rake spec

Credits: maxlinc@github gist).

Run a single spec / example

At first prepare the fixture environment:

bundle exec rake spec_prep

Then run in the bundle environment, run rspec on a specific spec:

bundle exec rspec spec/classes/someclass_spec.rb

Or you can filter based on the spec name:

bundle exec rspec --example mymodule::someclass

See rspec help for more details.

ruby debugger

You can use the gem pry to break on error and get shown a console in the context of the failure. To your Gemfile add gem 'pry' and install it with bundle install then to break inside a spec:

require 'spec_helper'
require 'pry'

describe 'mymodule::someclass' do

  it {
     # enable debugger
     binding.pry
     # compilation that fails: 
     should compile
  }
end

You will then be in a console before the breakage that let you inspect the environment (ls). See https://github.com/pry/pry for details.

reference

Integration with Jenkins

Note: as of January 25th patches have not been merged. See Gerrit topic:rspec-puppet.

Jenkins job simply runs rake test (CI entry point) from the root of the operations/puppet.git. The checks we want to run automatically are marked as prerequisites of the test task, for example:

task test: [:rubocop, :puppetlint_head, :syntax_head, :spec]

The tasks suffixed with _head are optimized to have the utility to only run on files changed in the proposed patch. Typically puppet-lint takes minutes to run against all the puppet manifests, when for CI we only are interested in the manifests that are actually being changed.

The rakefile add a task for each module having a spec directory. The task is named after the module and put under the namespace spec. Hence as soon as you create a basic structure for a module mymodule, you can run it from the root of the repository with:

bundle exec rake spec:mymodule

And it is dynamically made a prerequisite of the spec task which is run by CI. To say it otherwise, once a spec directory is created, Jenkins will try to run the spec.

As of January 25, patches are pending consensus. For the CI integration, at first the job will probably only be run on demand (by commenting check experimental in Gerrit). Notably the puppet catalog compilation is a bit slow (1.2sec per class) and we should run them in parallel.

Resources