httpbb

httpbb (for "HTTP Black Box") is a black-box testing tool for HTTP servers. It reads in a list of test cases, each of which is a specification for an HTTP request followed by one or more assertions about the response. Those requests are played against one or more servers, and the test passes if all the assertions come true.

The code is on Gerrit in the operations/software/httpbb repository.

Running tests

httpbb is installed on the deployment hosts and the Cumin hosts; the standard tests for app servers are installed into /srv/deployment/httpbb-tests. To run, pass one or more YAML test files (described below) on the command line, and one or more servers with the --hosts flag. For example, to run the full set of tests against all four mwdebug hosts,

rzl@cumin1001:~$ httpbb /srv/deployment/httpbb-tests/* --hosts=mwdebug[1001,1002].eqiad.wmnet,mwdebug[2001,2002].codfw.wmnet

(Tip: httpbb tests multiple hosts in parallel, but runs significantly faster when all hosts are in the local datacenter, due to the roundtrip latency for each request.)

Adding tests

The input to httpbb is one or more YAML files full of test cases. The app server tests are stored in modules/profile/files/httpbb in the Puppet repository.

The YAML has the following structure:

• Each top-level key consists of http:// or https:// followed by a host. The hostname given is moved into the Host header for outgoing requests. Multiple host names may be used in the same file, corresponding to virtual hosts on the same server. The actual destination for the traffic is given by the --hosts command-line flag.
• For each host key, the value is an array of test cases, expressed as mappings.
• Each test case includes a path field (e.g. /wiki/Main_Page) for the remainder of the outgoing request URL.
• Optionally, test cases may include a method (string). The default is GET.
• Optionally, test cases may include a request_headers (string mapping). The default is the Host header as described above, and an httpbb User-Agent. Test cases may override the User-Agent header.
• Optionally, test cases may include a form_body (string mapping) to set a request body shaped like POST-submitted form data. If present, httpbb URL-encodes the contents and adds the Content-Type: application/x-www-form-urlencoded header automatically. The HTTP method is not set automatically -- if you're using form_body, you probably also want to say method: POST.
• Each test case includes one or more assertion fields. If any assertion fails, the test fails. Supported assertions are:
  • assert_status (int) to check the status code of the response.
  • assert_headers (string mapping) to check one or more response headers. Header names are case-insensitive, and values must match exactly.
  • assert_headers_regex (string mapping) to check one or more response headers. Header names are case-insensitive, and values are partial-match regular expressions.
  • assert_headers_absent (string array) to assert that none of these (case-insensitive) header names are present in the response.
  • assert_body_contains (string) to check that a substring is present in the response text.
  • assert_body_regex (string) to check that the response text matches a multiline partial-match regular expression.

Examples

To assert that a server correctly serves the enwiki main page (or, at least, something with the correct title, along with an OK status):

https://en.wikipedia.org:
- path: /wiki/Main_Page
  assert_status: 200
  assert_body_contains: Wikipedia, the free encyclopedia

To assert that the default path / on Wiktionary serves a redirect to the main page, and that main page is served:

https://en.wiktionary.org:
- path: /
  assert_status: 301
  assert_headers:
    Location: https://en.wiktionary.org/wiki/Wiktionary:Main_Page
- path: /wiki/Wiktionary:Main_Page
  assert_status: 200
  assert_body_contains: Wiktionary, the free dictionary

(Since these tests share a virtual host, we don't spell out the key twice. This is a YAML mapping, so repeating a key would overwrite the previous value.)