You are browsing a read-only backup copy of Wikitech. The live site can be found at


From Wikitech-static
Revision as of 08:56, 14 October 2019 by imported>Giuseppe Lavagetto (→‎Building envoy for WMF infrastructure: Added details on how to create a new version.)
Jump to navigation Jump to search

What is envoy proxy

Envoy is an L7 proxy and communication bus designed for large modern service oriented architectures. It provides several features for a reverse proxy including but not limited to:

  • HTTP2 support.
  • L3/L4 filter architecture, so it can be used as a TLS terminator, traffic mirror service and other use cases.
  • Good observability and tracing, supporting statsd, zipking etc.
  • rate limiting, circuit breakers support.
  • dynamic configuration through the xDS protocol.
  • service discovery.
  • gRPC, Redis, MongoDB proxy support.

Envoy at WMF

There are two main use cases for envoy at WMF.

  • Act as a TLS terminator / proxy for internal services and potentially mediawiki.
  • Be deployed as a sidecar container to services running in the deployment pipeline and provide TLS termination, better observability, better logging for some services.

As more services move into the deployment pipeline, that is a kubernetes, these two use cases will converge into one.

Building envoy for WMF

Envoy community has presented recently an envoy proxy distribution that offers amongst other artifacts, when we started to consider envoy that distribution channel didn't exist at that time. Unfortunately, the deb packages they provide are quite incomplete.

Ship a new version

Clone the upstream envoy repository, and export an archive for the revision you want to package

$ export REF=v1.11.2  # use your own version here
$ git clone
$ pushd envoy && git archive --format=tar $REF | gzip > ../envoyproxy_$REF.orig.tar.gz
$ popd

Now clone operations/debs/envoyproxy, and import the tar archive you generated

$ mkdir WMF && cd WMF
$ USER="yourgerrituser" git clone "ssh://$" && scp -p -P 29418 $ "envoyproxy/.git/hooks/"
$ pushd envoyproxy
$ gbp import-orig ../../envoyproxy_$REF.orig.tar.gz
$ git push --tag
$ git push upstream

Now merge the new version in the wikimedia-stretch branch, create a new changelog entry, and push this as well.

Build the package on the WMF infrastructure

For building a new envoy debian package you should follow this steps.

  1. get access to the packaging project in Horizon, ask a project admin if you don't know who it is ask in #wikimedia-sre.
  2. ssh into builder-envoy.packaging.eqiad.wmflabs
  3. go to /tmp and clone the deb gerrit project.
  4. that repository includes the envoy source code and the debian control files. It has been created using gbp and using it is recommended. There is an upstream branch including the original source code from the GitHub repo and multiple upstream tags pointing to each imported version, a master branch that is the result of applying the latest upstream tag and possibly the development version of debian control files, after that several wikimedia-DIST branches which points to build packages to an specific distribution branch.
  5. The envoy building workflow is complex and involves running some docker containers and internet access, because of that this package cannot be build in our build serves. Providing that the debian control files are valid and can generate a proper debian package, go to /tmp/envoyproxy and execute sudo gbp buildpackage --git-builder='debuild -b -uc -us'. Most of the gbp options are preserved in debian/gbp.conf file. NOTE: giving the building requires docker pbuilder is not used.
  6. if the build process has been successful you should have a new debian package available that you can upload to the install servers.

Building new versions (minor or major)

  1. ssh into builder-envoy.packaging.eqiad.wmflabs
  2. get the tar.gz link for the version you want to build gbp asumes that you are always going to build a newer version, so be careful with that.
  3. execute gbp import-orig or the appropiate version, this will untar the file, update the upstream branch, create a tag for the new imported version and merge the mentioned upstream branch into master.
  4. checkout the wikimedia-DIST branch and merge from master or a concrete upstream tag.
  5. adapt or modify the debian control files to build the new version, when you have a working version commit and publish the changes in wikimedia-DIST branch.