You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Kubernetes/Ingress: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>JMeybohm
No edit summary
imported>BryanDavis
(→‎Configuration (for service owners): k8s-staging + port 30443 needed for staging)
 
(7 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Kubernetes nav}}
{{Kubernetes nav}}
{{Warn|content=Work in progress, treat this as a draft}}
== Introduction ==
== Introduction ==
The Kubernetes Ingress uses Istio Ingresscontroller (ultimately the Ingressgateway) running as a Daemonset on each worker node to route traffic to workload services and (in last instace) Pods.
The Kubernetes Ingress uses Istio Ingresscontroller (ultimately the Ingressgateway) running as a Daemonset on each worker node to route traffic to workload services and (in last instance) Pods.


The Istio Ingressgateway is implemented as an envoy instace that is configured via XDS (??) by the Istiod Control Plane. All configuration is derived from Kubernetes API Objects like the Service Objects as well as Istio specific custom resources: [https://istio.io/v1.9/docs/reference/config/networking/gateway/ Gateway], [https://istio.io/v1.9/docs/reference/config/networking/virtual-service/ VirtualService] and [https://istio.io/v1.9/docs/reference/config/networking/destination-rule/ DestinationRule].
The Istio Ingressgateway is implemented as an envoy instance that is configured via [https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol#xds-protocol xDS] by the Istiod Control Plane. All configuration is derived from Kubernetes API Objects like the Service Objects as well as Istio specific custom resources: [https://istio.io/v1.9/docs/reference/config/networking/gateway/ Gateway], [https://istio.io/v1.9/docs/reference/config/networking/virtual-service/ VirtualService] and [https://istio.io/v1.9/docs/reference/config/networking/destination-rule/ DestinationRule].


The process of configuring the Ingressgateway is abstracted away from service owners/deployers via helm {{Gitweb|project=operations/deployment_charts|file=common_templates/0.4/_ingress_helpers.tpl}} so that it is enough to specify <code>.Values.ingress.enabled: true</code> for a very basic setup.
The process of configuring the Ingressgateway is abstracted away from service owners/deployers via helm {{Gitweb|project=operations/deployment_charts|file=common_templates/0.4/_ingress_helpers.tpl}} so that it is enough to specify <code>.Values.ingress.enabled: true</code> for a very basic setup.
Line 12: Line 10:
* ingressgateway establishes TLS connections to upstream (pods) which will be terminated on pod level by service-proxy
* ingressgateway establishes TLS connections to upstream (pods) which will be terminated on pod level by service-proxy


== Setup and configuration ==
== Istio setup and configuration ==
Istio (the control plane as well as components like the Ingressgateway) are installed and initially configured using <code>istioctl</code> (from a deployment host) together with an environment specific config in {{Gitweb|project=operations/deployment_charts|file=custom_deploy.d/istio}}.
Istio (the control plane as well as components like the Ingressgateway) are installed and initially configured using <code>istioctl</code> (from a deployment host) together with an environment specific config in {{Gitweb|project=operations/deployment_charts|file=custom_deploy.d/istio}}.


Line 26: Line 24:
* [https://github.com/istio/istio/tree/1.9.5/manifests/charts Istio embedded helm charts]
* [https://github.com/istio/istio/tree/1.9.5/manifests/charts Istio embedded helm charts]


TLS certificates are generated and maintained by cert-manager and deployed into the istio-system namespace for the ingressgateway to pick them up. This is configured and deployed by SRE via helmfile.d/admin_ng/helmfile_namespace_certs.yaml ??. If a service needs to be made available under additional hostnames they need to be configured at as tlsHostname in the namespaces config (helmfile.d/admin_ng/values/*.yaml).
TLS certificates are generated and maintained by cert-manager and deployed into the istio-system namespace for the ingressgateway to pick them up. This is configured and deployed by SRE via <code>helmfile.d/admin_ng/helmfile_namespace_certs.yaml</code>.
== Troubleshooting ==
The Ingressgateway does emit access logs which can be viewed in [https://logstash.wikimedia.org/app/discover#/view/6c6f3990-932e-11ec-b1b4-8bb3c43a4085?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-15m%2Cto%3Anow)) logstash]
 
Generic grafana dashboards can be found via the istio tag: [https://grafana-rw.wikimedia.org/d/?tag=istio&search=open&orgId=1 Grafana search]
 
=== Status of the ingressgateway ===
Using 'proxy-status' it can be checked if the istio control-plane is successfully sending updates to the ingressgateway instances:
 
<syntaxhighlight lang="bash">
istioctl-1.9.5 proxy-status
</syntaxhighlight>
 
=== Retrieve clusters, listeners, routes, endpoints, secrets ===
istioctl can be used to fetch specific parts of the (envoy) configuration.
 
Be aware that ingressgateway will show a cluster and endpoint for every kubernetes service in the cluster, regardless of whether you have configured ingress for it or not (it's just discovering everything in the cluster).
 
<syntaxhighlight lang="bash">
# <COMPONENT> can be one of: clusters, endpoints, listeners, routes, secrets
istioctl-1.9.5 -n istio-system proxy-config <COMPONENT> daemonset/istio-ingressgateway
 
# You can output as JSON (-o json) for a huge amount of detail.
</syntaxhighlight>
 
When needing to troubleshoot TLS certificates used, the JSON output can be parsed with jq and piped into openssl, e.g:
<syntaxhighlight lang="bash">
# <RESOURCE NAME> is the name of the certificate as returned by:
istioctl-1.9.5 -n istio-system proxy-config secrets daemonset/istio-ingressgateway
 
# Dump a certificate chain
istioctl-1.9.5 -n istio-system proxy-config secrets daemonset/istio-ingressgateway -o json | \
  jq '[.dynamicActiveSecrets[] | select(.name == "<RESOURCE NAME>")][0].secret.tlsCertificate.certificateChain.inlineBytes' -r | \
  base64 -d | \
  openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -text -noout
 
# Dump a CA
istioctl-1.9.5 -n istio-system proxy-config secrets daemonset/istio-ingressgateway -o json | \
  jq '[.dynamicActiveSecrets[] | select(.name == "<RESOURCE NAME>")][0].secret.validationContext.trustedCa.inlineBytes' -r | \
  base64 -d | \
  openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -text -noout
</syntaxhighlight>
 
=== Dump complete envoy config ===
If needed, the complete (currently active) envoy config can be dumped from the ingressgateway:
<syntaxhighlight lang="bash">
kubectl -n istio-system exec -it daemonset/istio-ingressgateway -- \
  /bin/bash -c 'exec 5<>/dev/tcp/127.0.0.1/15000; echo -ne "GET /config_dump HTTP/1.1\r\nHost: localhost:15000\r\nConnection: close\r\n\r\n" >&5; cat <&5'
</syntaxhighlight>
 
See https://www.envoyproxy.io/docs/envoy/latest/operations/admin.html for details on interacting with the envoy admin interface.
 
You can always <code>kubectl -n istio-system port-forward daemonset/istio-ingressgateway 15000:15000</code> if you need to deal with it more.
 
=== Enable debug logging ===
Envoy component logging can be changed via istioctl as well:
<syntaxhighlight lang="bash">
# Print all components and the currently configured log level:
istioctl-1.9.5 -n istio-system proxy-config log daemonset/istio-ingressgateway
 
# Change to level of a specific component (admin in this case) to info
istioctl-1.9.5 -n istio-system proxy-config log daemonset/istio-ingressgateway --level admin:info
 
# Change the levels of all components to info
istioctl-1.9.5 -n istio-system proxy-config log daemonset/istio-ingressgateway --level info
</syntaxhighlight>
 
== Administration ==
 
=== Add a new service under Ingress ===
Assuming you want to add '''service-foo''' running in the '''main''' (wikikube) cluster under Ingress.
 
==== Configure certificates (optional) ====
By default Ingress will be configured with a certificate to terminate TLS for the hostnames:
 
* $NAMESPACE_NAME.discovery.wmnet
* $NAMESPACE_NAME.svc.codfw.wmnet
* $NAMESPACE_NAME.svc.eqiad.wmnet
 
If you have multiple user-facing services in your namespace or the hostnames need to derive from the default ($NAMESPACE_NAME), this can be configured using the <code>tlsHostnames</code> parameter to the namespace in <code>helmfile.d/admin_ng/values/main.yaml</code>. See docs at the start of {{Gitweb|project=operations/deployment-charts|file=helmfile.d/admin_ng/values/common.yaml}} for more details.
 
If additional SANs are needed that do not match the above schema (like '''service-foo.wikimedia.org'''), add those via the <code>tlsExtraSANs</code> parameter for your namespace in <code>helmfile.d/admin_ng/values/main.yaml</code>.
 
These changes need to be deployed to all main (wikikube) clusters like described in [[Kubernetes/Add a new service#Deploy changes to helmfile.d/admin ng]]
 
==== DNS changes ====
Your service will be made accessible by pointing dedicated CNAME records to the pre-existing LVS ''k8s-ingress-wikikube''.
 
* Add two datacenter specific CNAME records:
<syntaxhighlight lang=text>
; in $ORIGIN svc.eqiad.wmnet.
service-foo      1H  IN CNAME    k8s-ingress-wikikube.svc.eqiad.wmnet.
 
; in $ORIGIN svc.codfw.wmnet.
service-foo      1H  IN CNAME    k8s-ingress-wikikube.svc.codfw.wmnet.
</syntaxhighlight>
 
* If your service runs '''active/active:'''
<syntaxhighlight lang=text>
; in $ORIGIN discovery.wmnet.
service-foo      300 IN CNAME k8s-ingress-wikikube-ro.discovery.wmnet.
</syntaxhighlight>
 
* If your service runs '''active/passive:'''
<syntaxhighlight lang=text>
; in $ORIGIN discovery.wmnet.
service-foo      300 IN CNAME k8s-ingress-wikikube-rw.discovery.wmnet.
</syntaxhighlight>


= Debugging =
* Follow [[DNS#Changing records in a zonefile]] to create and deploy the zone
* istioctl commands to list proxy config etc.
* Run the <code>sre.dns.netbox</code> cookbook
* how/where to enable envoy debug logging (would be nice to figure out how to do that at runtime)
* Create a kibana dashboard for ingressgateway logs
* Create a grafana dashbaord for ingressgateway
* ?? Maybe we should disable generic access logging in ingressgateway?


==== Create an entry in the service::catalog ====
Add an stripped down entry for your service in {{gitweb|project=operations/puppet|file=hieradata/common/service.yaml}}, like:<syntaxhighlight lang="yaml">
  service-foo:
    description: Pretty fooish service, service-foo.svc.%{::site}.wmnet
    encryption: true
    ip: *k8s-ingress-wikikube_ips
    page: false
    probes: # monitoring for this service
      - type: http
    port: *k8s-ingress-wikikube_port
    sites:
      - eqiad
      - codfw
    state: service_setup
</syntaxhighlight>With that ''service::catalog'' entry, follow the state transitions like described in [[LVS#Create an entry in the service::catalog]] - switching to ''lvs_setup'' as well as puppet runs on icinga or auth-dns hosts can be skipped, as there is '''no lvs, discovery or monitoring stanza'''.
{{Warning|content=It's very likely that whenever puppet needs to run on A:icinga it actually needs to run on P{O:prometheus} instead}}


== Configuration (for service owners) ==
===Configuration (for service owners)===
To enable Ingress for your chart you need to undertake the following steps:
To enable Ingress for your chart you need to undertake the following steps:
* Make sure your chart uses the latest (at least 0.4) version of common_templates
*Make sure your chart uses the latest (at least 0.4) version of common_templates
* Make sure the {{Gitweb|project=operations/deployment_charts|file=common_templates/0.4/_ingress_helpers.tpl}}  is linked to the templates directory of your chart
*Make sure the {{Gitweb|project=operations/deployment_charts|file=common_templates/0.4/_ingress_helpers.tpl}}  is linked to the templates directory of your chart


For the absolute basic setup, all you need to do not is enable ingress via your values.yaml:
For the absolute basic setup, all you need to do not is enable ingress via your ''values.yaml'':
<syntaxhighlight lang="yaml">
<syntaxhighlight lang="yaml">
ingress:
ingress:
Line 48: Line 165:
</syntaxhighlight>
</syntaxhighlight>


This will make your service available as https://SERVICE_NAME.discovery.wmnet (https://SERVICE_NAME.staging.discovery.wmnet for staging) traffic will be routed as is to all pods of your service in a round-robin fashion.
This will make your service available as ''https<nowiki>:</nowiki>//SERVICE_NAME.discovery.wmnet'' (''https<nowiki>:</nowiki>//SERVICE_NAME.k8s-staging.discovery.wmnet:30443'' for staging) traffic will be routed as is to all pods of your service in a round-robin fashion.


You may configure more complex routing logic, listen to different or more than one hostname etc. via the ingress configuration stanza. Please keep in mind that for different or additional hostnames you may need SRE assistance to set up certificates etc.
You may configure more complex routing logic, listen to different or more than one hostname etc. via the ingress configuration stanza. Please keep in mind that for different or additional hostnames you may need SRE assistance to set up certificates etc.


The routing behavior may be modified via the ingress.httproutes stanza which supports all options described in https://istio.io/v1.9/docs/reference/config/networking/virtual-service/#HTTPRoute.
===More complex setups===
The routing behavior may be modified via the ''ingress.httproutes'' stanza which supports all options described in https://istio.io/v1.9/docs/reference/config/networking/virtual-service/#HTTPRoute.


If you want to make several services available as subpaths of a hostname (https://SERVICE_NAME.discovery.wmnet/foo, https://SERVICE_NAME.discovery.wmnet/bar, ...) you need to make sure to configure only one Istio Gateway (in one of your helm chart releases) for this hostname and attach multiple HTTPRoute objects to it. Multiple Istio Gateway objects claiming the same hostname will simply be ignored.
If you want to make several services available as subpaths of a hostname (https://SERVICE_NAME.discovery.wmnet/foo, https://SERVICE_NAME.discovery.wmnet/bar, ...) you need to make sure to configure only one Istio Gateway (in one of your helm chart releases) for this hostname and attach multiple HTTPRoute objects to it. Multiple Istio Gateway objects claiming the same hostname will simply be ignored.

Latest revision as of 20:12, 25 May 2022

Introduction

The Kubernetes Ingress uses Istio Ingresscontroller (ultimately the Ingressgateway) running as a Daemonset on each worker node to route traffic to workload services and (in last instance) Pods.

The Istio Ingressgateway is implemented as an envoy instance that is configured via xDS by the Istiod Control Plane. All configuration is derived from Kubernetes API Objects like the Service Objects as well as Istio specific custom resources: Gateway, VirtualService and DestinationRule.

The process of configuring the Ingressgateway is abstracted away from service owners/deployers via helm common_templates/0.4/_ingress_helpers.tpl so that it is enough to specify .Values.ingress.enabled: true for a very basic setup.

  • ingressgateway terminates TLS connections from clients
  • ingressgateway establishes TLS connections to upstream (pods) which will be terminated on pod level by service-proxy

Istio setup and configuration

Istio (the control plane as well as components like the Ingressgateway) are installed and initially configured using istioctl (from a deployment host) together with an environment specific config in custom_deploy.d/istio.

For initial installation or deploying updates, run (on a deployment host):

istioctl-1.9.5 apply -f <environment>/config.yaml

For details on how to configure Istio, please see:

Some parts of the configuration have to be (or can be) done via helm chart values. istioctl comes with embedded helm charts that get rendered and applied by istiotcl directly. You may find those embedded charts (to look up possible configuration options etc. at:

TLS certificates are generated and maintained by cert-manager and deployed into the istio-system namespace for the ingressgateway to pick them up. This is configured and deployed by SRE via helmfile.d/admin_ng/helmfile_namespace_certs.yaml.

Troubleshooting

The Ingressgateway does emit access logs which can be viewed in logstash

Generic grafana dashboards can be found via the istio tag: Grafana search

Status of the ingressgateway

Using 'proxy-status' it can be checked if the istio control-plane is successfully sending updates to the ingressgateway instances:

istioctl-1.9.5 proxy-status

Retrieve clusters, listeners, routes, endpoints, secrets

istioctl can be used to fetch specific parts of the (envoy) configuration.

Be aware that ingressgateway will show a cluster and endpoint for every kubernetes service in the cluster, regardless of whether you have configured ingress for it or not (it's just discovering everything in the cluster).

# <COMPONENT> can be one of: clusters, endpoints, listeners, routes, secrets
istioctl-1.9.5 -n istio-system proxy-config <COMPONENT> daemonset/istio-ingressgateway

# You can output as JSON (-o json) for a huge amount of detail.

When needing to troubleshoot TLS certificates used, the JSON output can be parsed with jq and piped into openssl, e.g:

# <RESOURCE NAME> is the name of the certificate as returned by:
istioctl-1.9.5 -n istio-system proxy-config secrets daemonset/istio-ingressgateway

# Dump a certificate chain
istioctl-1.9.5 -n istio-system proxy-config secrets daemonset/istio-ingressgateway -o json | \
  jq '[.dynamicActiveSecrets[] | select(.name == "<RESOURCE NAME>")][0].secret.tlsCertificate.certificateChain.inlineBytes' -r | \
  base64 -d | \
  openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -text -noout

# Dump a CA
istioctl-1.9.5 -n istio-system proxy-config secrets daemonset/istio-ingressgateway -o json | \
  jq '[.dynamicActiveSecrets[] | select(.name == "<RESOURCE NAME>")][0].secret.validationContext.trustedCa.inlineBytes' -r | \
  base64 -d | \
  openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -text -noout

Dump complete envoy config

If needed, the complete (currently active) envoy config can be dumped from the ingressgateway:

kubectl -n istio-system exec -it daemonset/istio-ingressgateway -- \
  /bin/bash -c 'exec 5<>/dev/tcp/127.0.0.1/15000; echo -ne "GET /config_dump HTTP/1.1\r\nHost: localhost:15000\r\nConnection: close\r\n\r\n" >&5; cat <&5'

See https://www.envoyproxy.io/docs/envoy/latest/operations/admin.html for details on interacting with the envoy admin interface.

You can always kubectl -n istio-system port-forward daemonset/istio-ingressgateway 15000:15000 if you need to deal with it more.

Enable debug logging

Envoy component logging can be changed via istioctl as well:

# Print all components and the currently configured log level:
istioctl-1.9.5 -n istio-system proxy-config log daemonset/istio-ingressgateway

# Change to level of a specific component (admin in this case) to info
istioctl-1.9.5 -n istio-system proxy-config log daemonset/istio-ingressgateway --level admin:info

# Change the levels of all components to info
istioctl-1.9.5 -n istio-system proxy-config log daemonset/istio-ingressgateway --level info

Administration

Add a new service under Ingress

Assuming you want to add service-foo running in the main (wikikube) cluster under Ingress.

Configure certificates (optional)

By default Ingress will be configured with a certificate to terminate TLS for the hostnames:

  • $NAMESPACE_NAME.discovery.wmnet
  • $NAMESPACE_NAME.svc.codfw.wmnet
  • $NAMESPACE_NAME.svc.eqiad.wmnet

If you have multiple user-facing services in your namespace or the hostnames need to derive from the default ($NAMESPACE_NAME), this can be configured using the tlsHostnames parameter to the namespace in helmfile.d/admin_ng/values/main.yaml. See docs at the start of helmfile.d/admin_ng/values/common.yaml for more details.

If additional SANs are needed that do not match the above schema (like service-foo.wikimedia.org), add those via the tlsExtraSANs parameter for your namespace in helmfile.d/admin_ng/values/main.yaml.

These changes need to be deployed to all main (wikikube) clusters like described in Kubernetes/Add a new service#Deploy changes to helmfile.d/admin ng

DNS changes

Your service will be made accessible by pointing dedicated CNAME records to the pre-existing LVS k8s-ingress-wikikube.

  • Add two datacenter specific CNAME records:
; in $ORIGIN svc.eqiad.wmnet.
service-foo       1H  IN CNAME    k8s-ingress-wikikube.svc.eqiad.wmnet.

; in $ORIGIN svc.codfw.wmnet.
service-foo       1H  IN CNAME    k8s-ingress-wikikube.svc.codfw.wmnet.
  • If your service runs active/active:
; in $ORIGIN discovery.wmnet.
service-foo       300 IN CNAME k8s-ingress-wikikube-ro.discovery.wmnet.
  • If your service runs active/passive:
; in $ORIGIN discovery.wmnet.
service-foo       300 IN CNAME k8s-ingress-wikikube-rw.discovery.wmnet.

Create an entry in the service::catalog

Add an stripped down entry for your service in hieradata/common/service.yaml, like:

  service-foo:
    description: Pretty fooish service, service-foo.svc.%{::site}.wmnet
    encryption: true
    ip: *k8s-ingress-wikikube_ips
    page: false
    probes: # monitoring for this service
      - type: http
    port: *k8s-ingress-wikikube_port
    sites:
      - eqiad
      - codfw
    state: service_setup

With that service::catalog entry, follow the state transitions like described in LVS#Create an entry in the service::catalog - switching to lvs_setup as well as puppet runs on icinga or auth-dns hosts can be skipped, as there is no lvs, discovery or monitoring stanza.

Configuration (for service owners)

To enable Ingress for your chart you need to undertake the following steps:

For the absolute basic setup, all you need to do not is enable ingress via your values.yaml:

ingress:
  enabled: true
  staging: true # If you are doing this for a staging service

This will make your service available as https://SERVICE_NAME.discovery.wmnet (https://SERVICE_NAME.k8s-staging.discovery.wmnet:30443 for staging) traffic will be routed as is to all pods of your service in a round-robin fashion.

You may configure more complex routing logic, listen to different or more than one hostname etc. via the ingress configuration stanza. Please keep in mind that for different or additional hostnames you may need SRE assistance to set up certificates etc.

More complex setups

The routing behavior may be modified via the ingress.httproutes stanza which supports all options described in https://istio.io/v1.9/docs/reference/config/networking/virtual-service/#HTTPRoute.

If you want to make several services available as subpaths of a hostname (https://SERVICE_NAME.discovery.wmnet/foo, https://SERVICE_NAME.discovery.wmnet/bar, ...) you need to make sure to configure only one Istio Gateway (in one of your helm chart releases) for this hostname and attach multiple HTTPRoute objects to it. Multiple Istio Gateway objects claiming the same hostname will simply be ignored.

Assuming you have two releases of your chart (one and two), you may configure ingress like in the following example to achieve that:

---
# release "one" values.yaml:
# made available as https://SERVICE_NAME.discovery.wmnet/one
# via default options + httproute
ingress:
  enabled: true
  httproutes:
  - match:
    - uri:
        prefix: /one
---
# release "two" values.yaml:
# made available as https://SERVICE_NAME.discovery.wmnet/two
# via the Gateway deployed by release "one"
ingress:
  enabled: true
  existingGateway: "SERVICE_NAMESPACE/one" # referencing the Gateway deployed by the release "one"
  routeHosts:
  - SERVICE_NAME.discovery.wmnet # Attach the following routes to this hostname in the referenced Gateway
  httproutes:
  - match:
    - uri:
        prefix: /two
    route:
    - destination:
      host: two-tls-service.SERVICE_NAMESPACE.svc.cluster.local # The cluster internal DNS name for this releases service
      port:
        number: SERVICE_TLS_PUBLIC_PORT # Port you defined in .Values.tls.public_port