Blubber

Blubber is a highly opinionated abstraction for container build configurations and a command-line compiler which currently supports outputting multi-stage Dockerfiles. It aims to provide a handful of declarative constructs that give developers control over build configurations without sacrificing security and maintainability.

Background

Blubber was initially developed to meet the build stage requirements of the Streamlined Service Delivery Design project (aka Release Pipeline, aka Continuous Delivery Pipeline). Initially it was thought that a developer might provide their own Dockerfile(s) for consumption by the pipeline. However, after some brief research and experimentation it was quickly apparent that writing an efficient and maintainable Dockerfile would require an inordinate degree of knowledge around layered filesystems, caching, directory context, obscure config format and inheritance, and unpredictable instruction behavior. In order to have sufficient trust in what was being tested—and what would eventually be deployed to production—Site Reliability Engineering, Release Engineering, and Services needed to adopt a better means of ingesting developer-provided image build configurations.

Release Engineering began experimenting with Blubber as a solution in early 2017 and officially took on the project in early fiscal year 2017-18 Q1 and continues to maintain and improve the project with support from SRE and Services.

Concepts

Declarative

Blubber provides developers with a simple YAML build configuration format for declaring: 1) what system dependencies their application requires; 2) what language-specific dependency manager to delegate to; 3) where the application files should be installed; 4) how the application needs to be tested; 5) how the application should run; 6) what variations of this configuration there need be for development, testing, and production (or any other) environments.

version: v3
base: docker-registry.wikimedia.org/nodejs-slim
apt: { packages: [librsvg2-2] }
lives:
  in: /srv/service

variants:
  build:
    base: docker-registry.wikimedia.org/nodejs-devel
    apt: { packages: [librsvg2-dev, git, pkg-config, build-essential] }
    node: { requirements: [package.json] }
    runs: { environment: { LINK: g++ } }
  test:
    includes: [build]
    entrypoint: [npm, test]

Stateless

Blubber runs as a stateless application, needing only the YAML configuration and a variant name for which to output a valid Dockerfile. It does not depend on anything else from the project filesystem or existing state of Docker images to function. Given a consistent configuration and variant name, its output is completely deterministic.

To demonstrate this, Blubber is currently running as a microservice (Blubberoid) on Toolforge, and can output a variant's Dockerfile via something like curl. For example, the above configuration piped to the below command would yield the "test" variant's Dockerfile.

curl -s --data-binary @- http://tools.wmflabs.org/blubber/test

Cache efficient

Blubber knows the idiosyncrasies of Docker's caching system and can produce consistently ordered Dockerfile output that makes full use of it. In addition to formatting and ordering instructions properly, it also knows how to delegate to package managers (e.g. Node's NPM, Python's Pip, etc.) in a way that will be cache efficient—package managers will only be re-run when building images if their related files are changed (e.g. package.json, requirements.txt, etc.).

Security focused

There's no built-in security model when writing raw Dockerfiles because it's assumed everything within your running container will be protected. This is simply false. Not all exploits are root exploits, and applications should still adhere to a sane security model for ownership and entry points so as to limit their attack surface and protect their runtime processes.

Blubber enforces a phased build process with dropped privileges and prevents users from inadvertently installing files as root or running their applications as a user that can write to application or system files.

Developer empowering

Blubber was designed with developer empowerment in mind. With the increased degree of trust afforded by its security model, Blubber can safely provide developers with configuration for defining all application dependencies, tests, and production entry points. And with a greater degree of trust in resulting images, Release Engineering and SRE can eventually provide developers with a more automated means of deployment.

Use

Blubber takes opinionated input files in YAML format and outputs a variant of a Dockerfiles that should be safe to run for their intended purpose. The number of potential variants in a Blubber config is not limited. Variants can inherit instructions, packages, and runtime environments from one another using the includes directive. They can also copy files from the resulting filesystem of another variant using the copies and artifacts directives.

Below is an example blubber.yaml for testing and running Mathoid.

version: v3
base: docker-registry.wikimedia.org/nodejs-slim
apt: { packages: [librsvg2-2] }
lives:
  in: /srv/service
runs:
  environment: { APP_BASE_PATH: /srv/service }

variants:
  build:
    base: docker-registry.wikimedia.org/nodejs-devel
    apt: { packages: [librsvg2-dev, git, pkg-config, build-essential] }
    node: { requirements: [package.json] }
    runs: { environment: { LINK: g++ } }
  development:
    includes: [build]
    entrypoint: [node, server.js]
  test:
    includes: [build]
    entrypoint: [npm, test]
  prep:
    includes: [build]
    node: { env: production }
  production:
    copies: prep
    node: { env: production }
    entrypoint: [node, server.js]

Installation

APT

Blubber is available from our APT repo.

deb http://apt.wikimedia.org/wikimedia stretch-wikimedia main

apt-get install blubber

Binaries

Blubber binary releases are currently available for:

• OSX
• Windows
• Linux amd64/386/arm/arm64
• And, of course, plan9

Microservice

Blubber is currently running on Toolforge as a microservice called Blubberoid. See #Stateless.

Source

You can also install blubber from source. Blubber requires go >= 1.9 (>=1.10 recommended) and related tools

• To install on rpm style systems: sudo dnf install golang golang-godoc
• To install on apt style systems: sudo apt install golang golang-golang-x-tools
• To install on macOS use Homebrew and run: brew install go
• You can run go version to check the golang version.
• If your distro's go package is too old or unavailable, download a newer golang version.

$ export GOPATH="$HOME/go"
$ go get phabricator.wikimedia.org/source/blubber

This will install blubber to ~/go/bin/blubber

Example usage

In order to output a Dockerfile, Blubber needs a configuration file and a variant that you want to output. The available variants in the file above are build, development, test, prep, and production. To run current tests in a test build using blubber pass the test variant to Docker to build and run. For example in the Mathoid repository, the Blubber configuration file lives under the path dist/pipeline/blubber.yaml:

$ cd ~/src/mathoid
$ blubber dist/pipeline/blubber.yaml test | docker build -t mathoid-test-$(date --iso) -f - .
$ docker run --rm -it mathoid-test-$(date --iso)
...
$ docker rmi mathoid-test-$(date --iso)