You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Difference between revisions of "Envoy"

From Wikitech-static
Jump to navigation Jump to search
imported>RLazarus
(→‎Ship a new version: update procedure for repack script)
imported>Giuseppe Lavagetto
Line 15: Line 15:
There are two main use cases for envoy at WMF.
There are two main use cases for envoy at WMF.


* Act as a TLS terminator / proxy for internal services and potentially mediawiki.
* Act as a TLS terminator / proxy for internal services. This is done for services:
* Be deployed as a sidecar container to services running in the deployment pipeline and provide TLS termination, better observability, better logging for some services.
** in the deployment pipeline (via the [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/common_templates/0.1/_tls_helpers.tpl tls helpers] in the deployment charts) where it works as a sidecar container to the service if tls is enabled for the specific chart.
** For services not in the pipeline, using <tt>profile::tlsproxy::envoy</tt>
* Act as a local proxy to other services for MediaWiki (for now), via <tt>profile::services_proxy::envoy</tt>


As more services move into the deployment pipeline, that is a kubernetes, these two use cases will converge into one.
=== TLS termination ===
If you want to add TLS termination to a new deployment chart, just use the scaffold script - it will create your starting chart with tls termination primitives already in place.
If you want to add TLS termination to an existing chart, you just have to:
* Link <tt>common_templates/<version>/_tls_helpers.tpl</tt> in the <tt>templates</tt> directory of the chart
* Insert the appropriate calls to those templates across the configmap, deployment, service and networkpolicy templates.
 
See https://gerrit.wikimedia.org/r/#/c/operations/deployment-charts/+/558092/ as an example.
 
If you want to add TLS termination to a service in puppet, include <tt>profile::tlsproxy::envoy</tt> in its role in puppet, and add the hiera configuration following the suggestions in the class documentation.
 
=== Services Proxy ===
The services proxy is installed on all servers that run MediaWiki, and does expose them via HTTP on <tt>localhost:<PORT></tt>. Some endpoints might also define a specific <tt>Host</tt> header.
The service proxy offers:
* Persistent connections
* Advanced TLS tunneling (envoy supports TLS 1.3)
* Retry logic
* Circuit breaking (still not implemented)
* Header rewriting
* Telemetry for all backends
* Tracing (still not implemented)
* Precise timeouts (microsecond resolution)
The currently defined services are defined in <tt>hieradata/common/profile/services_proxy/envoy.yaml</tt>. To add a new service you just need to add an entry to that list. Please refer to the class documentation in puppet for details. You can define your proxy to point to any valid DNS record, which will be re-resolved periodically. This means, it works with discovery records in DNS.


== Building envoy for WMF ==
== Building envoy for WMF ==
Line 24: Line 47:
Envoy community has presented recently https://www.getenvoy.io/ an envoy proxy distribution that offers amongst other artifacts, when we started to consider envoy that distribution channel didn't exist at that time. Unfortunately, the deb packages they provide are quite incomplete.
Envoy community has presented recently https://www.getenvoy.io/ an envoy proxy distribution that offers amongst other artifacts, when we started to consider envoy that distribution channel didn't exist at that time. Unfortunately, the deb packages they provide are quite incomplete.


=== Ship a new version ===
=== Prepare a new version ===


The ''operations/debs/envoyproxy'' repository includes the envoy source code and the debian control files. It has been created using [[debian:PackagingWithGit|gbp]] and using it is recommended. There is an upstream branch including the original source code from the GitHub repo and multiple upstream tags pointing to each imported version, and a master branch that is the result of applying the latest upstream tag and possibly the development version of debian control files.
The ''operations/debs/envoyproxy'' repository includes the envoy source code and the debian control files. It has been created using [[debian:PackagingWithGit|gbp]] and using it is recommended. There is an upstream branch including the original source code from the GitHub repo and multiple upstream tags pointing to each imported version, and a master branch that is the result of applying the latest upstream tag and possibly the development version of debian control files.
Line 67: Line 90:
# The process leaves behind a 100+-GB artifact, which you should clean up: <code>rm /tmp/envoy-docker-build</code> (If you need to rebuild for any reason, leave the file in place: the build will use it to run incrementally if appropriate, and will complete much faster.)
# The process leaves behind a 100+-GB artifact, which you should clean up: <code>rm /tmp/envoy-docker-build</code> (If you need to rebuild for any reason, leave the file in place: the build will use it to run incrementally if appropriate, and will complete much faster.)
# Copy the output files from <code>/usr/src</code> to install1002.eqiad.wmnet.
# Copy the output files from <code>/usr/src</code> to install1002.eqiad.wmnet.
=== Update envoy in production ===
==== Physical hosts ====
Just use [[Debdeploy]] as usual. It is advised that a new version is rolled out as follows:
* On the mediawiki canaries, and on the restbase canaries
* On the rest of tls-terminated proxies
* On the rest of MediaWiki
==== Kubernetes/deployment pipeline ====
For services running on kubernetes, the procedure is a bit more complex at the time of writing:
* Bump the changelog of both the <tt>envoy</tt> image and the <tt>envoy-tls-proxy</tt> images, see https://gerrit.wikimedia.org/r/#/c/operations/docker-images/production-images/+/581747/ for example
* Go on one build server (role <tt>role::builder</tt> in puppet) and run
<syntaxhighlight lang="bash">
$ cd /srv/images/production-images
# If someone's been naughty and hand patched the repo, this will alert you before messing with the local git history
$ sudo git pull --ff-only
$ sudo build-production-images
</syntaxhighlight>
The script will only build the images not present on our the docker registry - so in your case supposedly only the envoy images.

Revision as of 12:01, 20 March 2020

What is envoy proxy

Envoy is an L7 proxy and communication bus designed for large modern service oriented architectures. It provides several features for a reverse proxy including but not limited to:

  • HTTP2 support.
  • L3/L4 filter architecture, so it can be used as a TLS terminator, traffic mirror service and other use cases.
  • Good observability and tracing, supporting statsd, zipking etc.
  • rate limiting, circuit breakers support.
  • dynamic configuration through the xDS protocol.
  • service discovery.
  • gRPC, Redis, MongoDB proxy support.

Envoy at WMF

There are two main use cases for envoy at WMF.

  • Act as a TLS terminator / proxy for internal services. This is done for services:
    • in the deployment pipeline (via the tls helpers in the deployment charts) where it works as a sidecar container to the service if tls is enabled for the specific chart.
    • For services not in the pipeline, using profile::tlsproxy::envoy
  • Act as a local proxy to other services for MediaWiki (for now), via profile::services_proxy::envoy

TLS termination

If you want to add TLS termination to a new deployment chart, just use the scaffold script - it will create your starting chart with tls termination primitives already in place. If you want to add TLS termination to an existing chart, you just have to:

  • Link common_templates/<version>/_tls_helpers.tpl in the templates directory of the chart
  • Insert the appropriate calls to those templates across the configmap, deployment, service and networkpolicy templates.

See https://gerrit.wikimedia.org/r/#/c/operations/deployment-charts/+/558092/ as an example.

If you want to add TLS termination to a service in puppet, include profile::tlsproxy::envoy in its role in puppet, and add the hiera configuration following the suggestions in the class documentation.

Services Proxy

The services proxy is installed on all servers that run MediaWiki, and does expose them via HTTP on localhost:<PORT>. Some endpoints might also define a specific Host header. The service proxy offers:

  • Persistent connections
  • Advanced TLS tunneling (envoy supports TLS 1.3)
  • Retry logic
  • Circuit breaking (still not implemented)
  • Header rewriting
  • Telemetry for all backends
  • Tracing (still not implemented)
  • Precise timeouts (microsecond resolution)

The currently defined services are defined in hieradata/common/profile/services_proxy/envoy.yaml. To add a new service you just need to add an entry to that list. Please refer to the class documentation in puppet for details. You can define your proxy to point to any valid DNS record, which will be re-resolved periodically. This means, it works with discovery records in DNS.

Building envoy for WMF

Envoy community has presented recently https://www.getenvoy.io/ an envoy proxy distribution that offers amongst other artifacts, when we started to consider envoy that distribution channel didn't exist at that time. Unfortunately, the deb packages they provide are quite incomplete.

Prepare a new version

The operations/debs/envoyproxy repository includes the envoy source code and the debian control files. It has been created using gbp and using it is recommended. There is an upstream branch including the original source code from the GitHub repo and multiple upstream tags pointing to each imported version, and a master branch that is the result of applying the latest upstream tag and possibly the development version of debian control files.

Clone the debs repo and use its debian/repack script to clone the upstream envoy repository and export an archive for the revision you want to package. Run the script from outside the repo, in order to avoid dirtying the working directory with the archive.

$ export REF=v1.11.2  # use your own version here
$ USER="yourgerrituser" git clone "ssh://$USER@gerrit.wikimedia.org:29418/operations/debs/envoyproxy" && scp -p -P 29418 $USER@gerrit.wikimedia.org:hooks/commit-msg "envoyproxy/.git/hooks/"
$ envoyproxy/debian/repack $REF

Now, inside the repo, import the tar archive you generated.

$ cd envoyproxy
$ git branch upstream && git branch -u origin/upstream upstream
$ gbp import-orig ../../envoyproxy_$REF.orig.tar.gz
$ git push origin upstream
$ git push --tag

Now create a new changelog entry on master, and push this as well.

$ export DEBEMAIL="$(git config --get user.name) <$(git config --get user.email)>"
$ dch -v ${REF#v}-1 -D buster-wikimedia --force-distribution "New upstream version ${REF#v}"
$ git commit debian/changelog -m "New upstream version ${REF#v}"
$ git push

Build the package on the WMF infrastructure

For building a new envoy debian package you should follow this steps.

  1. get access to the packaging project in Horizon, ask a project admin if you don't know who it is ask in #wikimedia-sre.
  2. Add your ssh public key (not the same one you use for production) under Preferences > OpenStack on Wikitech.
  3. ssh into builder-envoy.packaging.eqiad.wmflabs
  4. go to /usr/src/envoyproxy and pull the master branch and upstream tags: git checkout master && git fetch --tags && git pull --force --rebase
  5. Run the /mnt/build_envoy.sh $DISTRO script, where $DISTRO should be the distribution code name.
  6. The envoy building workflow is complex and involves running some docker containers and internet access, because of that this package cannot be build in our build servers. It uses a patched-up version of what pbuilder does, just done manually.
  7. If the build process goes well, your new packages will be under /usr/src
  8. The process leaves behind a 100+-GB artifact, which you should clean up: rm /tmp/envoy-docker-build (If you need to rebuild for any reason, leave the file in place: the build will use it to run incrementally if appropriate, and will complete much faster.)
  9. Copy the output files from /usr/src to install1002.eqiad.wmnet.

Update envoy in production

Physical hosts

Just use Debdeploy as usual. It is advised that a new version is rolled out as follows:

  • On the mediawiki canaries, and on the restbase canaries
  • On the rest of tls-terminated proxies
  • On the rest of MediaWiki

Kubernetes/deployment pipeline

For services running on kubernetes, the procedure is a bit more complex at the time of writing:

$ cd /srv/images/production-images
# If someone's been naughty and hand patched the repo, this will alert you before messing with the local git history
$ sudo git pull --ff-only
$ sudo build-production-images

The script will only build the images not present on our the docker registry - so in your case supposedly only the envoy images.