You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Blubber: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Lars Wirzenius
imported>BearND
(Need to get blubber before using it)
 
(5 intermediate revisions by 3 users not shown)
Line 1: Line 1:
[[File:Blubber Logo.png|thumb]]
[[Image:Blubber logo.svg|300px|frameless|center|middle]]


Blubber is a highly opinionated abstraction for container build configurations and a command-line compiler which currently supports outputting multi-stage Dockerfiles. It aims to provide a handful of declarative constructs that give developers control over build configurations without sacrificing security and maintainability.
<templatestyles src="Template:FancyDoc/style.css" />
<div class="fancydoc-header fancydoc--green-border">
Blubber creates Dockerfiles from a higher level description expressed as YAML. Blubber is a highly opinionated abstraction for container build configurations and a command-line compiler which currently supports outputting [https://docs.docker.com/develop/develop-images/multistage-build/ multi-stage] Dockerfiles.


== Background ==
It aims to provide a handful of declarative constructs that accomplish build configuration in a more secure and deterministic way than running ad-hoc commands.


Blubber was initially developed to meet the build stage requirements of the [[Streamlined Service Delivery Design]] project (aka Release Pipeline, aka Continuous Delivery Pipeline). Initially it was thought that a developer might provide their own <code>Dockerfile</code>(s) for consumption by the pipeline. However, after some brief research and experimentation it was quickly apparent that writing an efficient and maintainable <code>Dockerfile</code> would require an inordinate degree of knowledge around layered filesystems, caching, directory context, obscure config format and inheritance, and unpredictable instruction behavior. In order to have sufficient trust in what was being tested—and what would eventually be deployed to production—[[mw:Wikimedia_Site_Reliability_Engineering|Site Reliability Engineering]], [[mw:Wikimedia_Release_Engineering_Team|Release Engineering]], and [[mw:Wikimedia_Services|Services]] needed to adopt a better means of ingesting developer-provided image build configurations.
Blubber is used by Wikimedia CI to test code and publish production-ready Docker images.
</div>


Release Engineering began experimenting with Blubber as a solution in early 2017 and officially took on the project in early fiscal year 2017-18 Q1 and continues to maintain and improve the project with support from SRE and Services.
<templatestyles src="Template:Portal_list_item/styles.css" />
<div class="mw-tpl-portal-list"><div class="mw-tpl-portal-list-caption">Documentation</div>
{{Portal list item noimage
|name=Getting Blubber
|link=Blubber/Download
|description=Download the Blubber command line application
}}
{{Portal list item noimage
|name=Getting started
|link=Blubber/Tutorial
|description=Try a walkthrough that helps you create your first <code>Blubberfile</code>, create your first Docker images, and start using Blubber for your project
}}
{{Portal list item noimage
|name=Use in the Wikimedia Pipeline
|link=Blubber/Pipeline
|description=Wikimedia's [[Deployment pipeline]] pipeline project makes use of Blubber to run tests and publish images.
}}
{{Portal list item noimage
|name=Concepts
|link=Blubber/Idea
|description=Blubber's background, philosophy, and implementation
}}
{{Portal list item noimage
|name=User Guide
|link=Blubber/User Guide
|description=Detailed Blubber User Guide
}}
</div>


== Concepts ==
== Developers ==


=== Declarative ===
There is a short guide to [https://gerrit.wikimedia.org/g/blubber/+/master/CONTRIBUTING.md contributing to Blubber] for developers in the [https://gerrit.wikimedia.org/g/blubber Blubber repository] on Gerrit.
 
Blubber provides developers with a simple YAML build configuration format for declaring: 1) what system dependencies their application requires; 2) what language-specific dependency manager to delegate to; 3) where the application files should be installed; 4) how the application needs to be tested; 5) how the application should run; 6) what variations of this configuration there need be for development, testing, and production (or any other) environments.
 
<syntaxhighlight lang="yaml">
version: v3
base: docker-registry.wikimedia.org/nodejs-slim
apt: { packages: [librsvg2-2] }
lives:
  in: /srv/service
 
variants:
  build:
    base: docker-registry.wikimedia.org/nodejs-devel
    apt: { packages: [librsvg2-dev, git, pkg-config, build-essential] }
    node: { requirements: [package.json] }
    runs: { environment: { LINK: g++ } }
  test:
    includes: [build]
    entrypoint: [npm, test]
</syntaxhighlight>
 
=== Stateless ===
 
Blubber runs as a stateless application, needing only the YAML configuration and a variant name for which to output a valid <code>Dockerfile</code>. It does not depend on anything else from the project filesystem or existing state of Docker images to function. Given a consistent configuration and variant name, its output is completely deterministic.
 
To demonstrate this, Blubber is currently running as a microservice ([https://toolsadmin.wikimedia.org/tools/id/blubber Blubberoid]) on [[Toolforge]], and can output a variant's <code>Dockerfile</code> via something like <code>curl</code>. For example, the above configuration piped to the below command would yield the "test" variant's <code>Dockerfile</code>.
 
<syntaxhighlight lang="bash">
curl -s --data-binary @- http://tools.wmflabs.org/blubber/test
</syntaxhighlight>
 
=== Cache efficient ===
 
Blubber knows the idiosyncrasies of Docker's caching system and can produce consistently ordered <code>Dockerfile</code> output that makes full use of it. In addition to formatting and ordering instructions properly, it also knows how to delegate to package managers (e.g. Node's NPM, Python's Pip, etc.) in a way that will be cache efficient—package managers will only be re-run when building images if their related files are changed (e.g. <code>package.json</code>, <code>requirements.txt</code>, etc.).
 
=== Security focused ===
 
There's no built-in security model when writing raw <code>Dockerfile</code>s because it's assumed everything within your running container will be protected. This is simply false. Not all exploits are root exploits, and applications should still adhere to a sane security model for ownership and entry points so as to limit their attack surface and protect their runtime processes.
 
Blubber enforces a phased build process with dropped privileges and prevents users from inadvertently installing files as root or running their applications as a user that can write to application or system files.
 
=== Developer empowering ===
 
Blubber was designed with developer empowerment in mind. With the increased degree of trust afforded by its security model, Blubber can safely provide developers with configuration for defining all application dependencies, tests, and production entry points. And with a greater degree of trust in resulting images, Release Engineering and SRE can eventually provide developers with a more automated means of deployment.
 
== Use ==
 
Blubber takes opinionated input files in YAML format and outputs a variant of a <code>Dockerfile</code>s that should be safe to run for their intended purpose. The number of potential variants in a Blubber config is not limited. Variants can inherit instructions, packages, and runtime environments from one another using the <code>includes</code> directive. They can also copy files from the resulting filesystem of another variant using the <code>copies</code> and <code>artifacts</code> directives.
 
Below is an example <code>blubber.yaml</code> for testing and running [[Mathoid]].
 
<syntaxhighlight lang="yaml">
version: v3
base: docker-registry.wikimedia.org/nodejs-slim
apt: { packages: [librsvg2-2] }
lives:
  in: /srv/service
runs:
  environment: { APP_BASE_PATH: /srv/service }
 
variants:
  build:
    base: docker-registry.wikimedia.org/nodejs-devel
    apt: { packages: [librsvg2-dev, git, pkg-config, build-essential] }
    node: { requirements: [package.json] }
    runs: { environment: { LINK: g++ } }
  development:
    includes: [build]
    entrypoint: [node, server.js]
  test:
    includes: [build]
    entrypoint: [npm, test]
  prep:
    includes: [build]
    node: { env: production }
  production:
    copies: prep
    node: { env: production }
    entrypoint: [node, server.js]
</syntaxhighlight>
 
== Installation ==
 
=== APT ===
 
Blubber is available from our [[APT]] repo.
 
<syntaxhighlight lang="bash">
deb http://apt.wikimedia.org/wikimedia stretch-wikimedia main
</syntaxhighlight>
 
<syntaxhighlight lang="bash">
apt-get install blubber
</syntaxhighlight>
 
=== Binaries ===
 
[https://people.wikimedia.org/~thcipriani/blubber/ Blubber binary releases] are currently available for:
 
* [https://people.wikimedia.org/~thcipriani/blubber/darwin-amd64 OSX]
* [https://people.wikimedia.org/~thcipriani/blubber/windows-amd64 Windows]
* Linux [https://people.wikimedia.org/~thcipriani/blubber/linux-amd64 amd64]/[https://people.wikimedia.org/~thcipriani/blubber/linux-386 386]/[https://people.wikimedia.org/~thcipriani/blubber/linux-arm arm]/[https://people.wikimedia.org/~thcipriani/blubber/linux-arm64 arm64]
* And, of course, [https://people.wikimedia.org/~thcipriani/blubber/plan9-amd64 plan9]
 
=== Microservice ===
 
Blubber is currently running on [[Toolforge]] as a microservice called [https://toolsadmin.wikimedia.org/tools/id/blubber Blubberoid]. See [[#Stateless]].
 
=== Source ===
 
You can also install blubber from source. Blubber requires <code>go</code> >= 1.9 (>=1.10 recommended) and related tools
 
* To install on rpm style systems: <code>sudo dnf install golang golang-godoc</code>
* To install on apt style systems: <code>sudo apt install golang golang-golang-x-tools</code>
* To install on macOS use [https://brew.sh Homebrew] and run: <code>brew install go</code>
* You can run <code>go version</code> to check the golang version.
* If your distro's go package is too old or unavailable, [https://golang.org/dl/ download] a newer golang version.                                             
 
<syntaxhighlight lang="bash">
$ export GOPATH="$HOME/go"
$ go get phabricator.wikimedia.org/source/blubber
</syntaxhighlight>
 
This will install blubber to <code>~/go/bin/blubber</code>
 
== Example usage ==
 
In order to output a <code>Dockerfile</code>, Blubber needs a configuration file and a variant that you want to output. The available variants in the file above are <code>build</code>, <code>development</code>, <code>test</code>, <code>prep</code>, and <code>production</code>. To run current tests in a test build using blubber pass the test variant to <code>Docker</code> to build and run. For example in the Mathoid repository, the Blubber configuration file lives under the path <code>dist/pipeline/blubber.yaml</code>:
 
<syntaxhighlight lang="bash">
$ cd ~/src/mathoid
$ blubber dist/pipeline/blubber.yaml test | docker build -t mathoid-test-$(date --iso) -f - .
$ docker run --rm -it mathoid-test-$(date --iso)
...
$ docker rmi mathoid-test-$(date --iso)
</syntaxhighlight>
 
 
= User Guide =
 
'''Disclaimer!''' This is the beautiful and perfect and absolutely correct
and error-free documentation for Blubber. If reality disagrees,
reality is unambiguously wrong and needs to be replaced. (Not really.
Please update and fix this wiki page instead.)
 
Blubber produces Dockerfiles for bulding Docker images. It is written
and maintained by the [[mw:Wikimedia_Release_Engineering_Team|Release Engineering]] team
as part of a toolset for delivering software into
production, testers, developers, and automated testing in CI.
 
Blubber reads a higher-level specification and writes the
corresponding Dockerfile. Docker tooling can use that to build a
container image. The image can then be used to run an application, or
deployed to Kubernetes to run a container, for development purposes,
or in production. The image can also be shared with anyone who wants
to run the application on their own, perhaps to test, debug, or
develop it.
 
Blubber runs on its own. It is not directly connected to WMF
infrastructure. Blubber does not run Docker itself.
 
== Example ==
 
<source lang="yaml">
---
version: v3
base: debian:jessie
apt:
  packages: [libjpeg, libyaml]
python:
  version: python2.7
runs:
  environment:
    FOO: bar
    BAR: baz
 
variants:
  build:
    apt:
      packages: [libjpeg-dev, libyaml-dev]
    node:
      requirements: [package.json, package-lock.json]
    python:
      requirements: [requirements.txt]
    builder:
      command: [make, deps]
      requirements: [Makefile, vendor]
 
  development:
    includes: [build]
 
test:
    includes: [build]
    apt:
      packages: [chromium]
    python:
      requirements: [requirements.txt, test-requirements.txt, docs/requirements.txt]
    runs:
      insecurely: true
    entrypoint: [npm, test]
 
  prep:
    includes: [build]
    node:
      env: production
 
  production:
    base: debian:jessie-slim
    node:
      env: production
    copies: prep
    entrypoint: [node, server.js]
</source>
 
To run Blubber, use a command like the following:
 
    blubber blubber.example.yaml test > Dockerfile
 
The output will look something like the following (the details may
vary, and probably will):
 
    FROM debian:jessie
    USER "root"
    ENV DEBIAN_FRONTEND="noninteractive"
    RUN apt-get update && apt-get install -y "libjpeg" "libyaml" "libjpeg-dev" "libyaml-dev" && rm -rf /var/lib/apt/lists/*
    RUN python2.7 "-m" "easy_install" "pip" && python2.7 "-m" "pip" "install" "-U" "setuptools" "wheel" "tox"
    LABEL blubber.variant="build" blubber.version="0.4.0+882f0fc"
 
To build the Docker image, run the following command:
 
    docker build -t blubbered .
 
(Assuming the Dockerfile you created by running blubber is in the
current direcory.) To run the image:
 
    docker run --rm -p 4000:8080 blubbered python3 /srv/service/hello.py
 
(FIXME: The above doesn't do anything useful for the example.)
 
You can publish the image to a Docker registry or share it with others
in the usual way for Docker images. Or you can share only the
Dockerfile, or the Blubber specification file, depending on what's
best for you and your collaborators.
 
== Building and installing Blubber ==
 
Blubber is written in the Go programming language. To build and
install it, you need to install the Go toolchain. On a Debian or
Debian-based system:
 
    sudo apt install golang-go
    mkdir ~/go
    export GOPATH="$HOME/go"
 
To fetch the source code:
 
    go get gerrit.wikimedia.org/r/blubber
 
At the time of writing, the above will fail with an error about there
being no Go source files, but you can ignore the failure. It's benign.
 
    cd ~/go/src/gerrit.wikimedia.org/r/blubber
    make
 
You now have the executable:
 
    ~/go/bin/blubber
 
If you don't want to type the whole path to the executable, you can
add `~/go/bin` go your `PATH` environment variable, or copy the
executable somewhere that is already in PATH.
 
 
== Blubber specification files ==
 
Blubber specification files use the YAML format. The top level is
a "dict", or a set of key/value pairs. The toplevel items (keys) that
Blubber knows about are:
 
; version : the version of the Blubber specification file syntax; use ''v3'' or Blubber will be very upset indeed.
 
; base : the base image upon which the new Docker image will be built; ''docker-registry.wikimedia.org/wikimedia-stretch:latest'' is a good starting point for WMF
 
; apt : additional Debian packages to install with the ''apt'' command; the value should be a list of package names
 
; python : which version of Python to use; the value should be ''version: python2.7 or similar; FIXME: what does this do?
 
; runs : settings for things run in the container, value should be a dict, which can have the following keys:
 
:; environment : environment variables to set
 
:; insecurely : Boolean value: should the application in the container be run as a user that can't write anything to the filesystem, including caches, or as a user that can? Production variants should have this set to `false`, but other variants mayset it to `true` to make things easier.
 
; node : FIXME
 
; builder : FIXME
 
; entrypoint : FIXME
 
; variants : value should be a dict, whose keys name the variants that can be built; see below for details
 
== Variants ==
 
Blubber can build several variants of an image from the same
specification file. The variants are named and described under the
'''variants''' top level item. Typically, there are variants for
development versus production: the development variant might have more
debugging tools, for example, which are not included in the production
variant, which should have no extra software installed to minimize
risk of security problems, or other problems.
 
A variant is built using the top level items, combined with the items
for the variant. So if the top level '''apt''' installed some packages,
and the variant's '''apt''' some other packages, both sets of packages get
installed in that variant.
 
Variants can have the fields that are valid at the top level, plus
additionally:
 
; includes : base this variant on the top level plus all the variants named in this field

Latest revision as of 19:22, 6 May 2020

Blubber logo.svg

Blubber creates Dockerfiles from a higher level description expressed as YAML. Blubber is a highly opinionated abstraction for container build configurations and a command-line compiler which currently supports outputting multi-stage Dockerfiles.

It aims to provide a handful of declarative constructs that accomplish build configuration in a more secure and deterministic way than running ad-hoc commands.

Blubber is used by Wikimedia CI to test code and publish production-ready Docker images.

Documentation
Download the Blubber command line application
Try a walkthrough that helps you create your first Blubberfile, create your first Docker images, and start using Blubber for your project
Wikimedia's Deployment pipeline pipeline project makes use of Blubber to run tests and publish images.
Blubber's background, philosophy, and implementation
Detailed Blubber User Guide

Developers

There is a short guide to contributing to Blubber for developers in the Blubber repository on Gerrit.