imported>JMeybohm |
imported>Btullis |
| (41 intermediate revisions by 17 users not shown) |
| Line 1: |
Line 1: |
| | {{Kubernetes nav}} |
| :''For information about Kubernetes in the Toolforge environment see [[Help:Toolforge/Kubernetes]].'' | | :''For information about Kubernetes in the Toolforge environment see [[Help:Toolforge/Kubernetes]].'' |
| | |
| '''[[w:Kubernetes|Kubernetes]]''' (often abbreviated '''k8s''') is an open-source system for automating deployment, and management of applications running in [[W:Operating-system-level virtualization|containers]]. This page collects some notes/docs on the Kubernetes setup in the Foundation production environment. | | '''[[w:Kubernetes|Kubernetes]]''' (often abbreviated '''k8s''') is an open-source system for automating deployment, and management of applications running in [[W:Operating-system-level virtualization|containers]]. This page collects some notes/docs on the Kubernetes setup in the Foundation production environment. |
| | |
| | == Clusters == |
| | |
| | We maintain Kubernetes clusters in both the [[SRE/Production access|production]] and the [[Help:Cloud Services introduction|cloud services]] realms. |
| | |
| | Most of the information on this page and its subpages applies to the clusters in the production realm, although some techniques and tools are broadly applicable to other WMF clusters and Kubernetes in general. |
| | |
| | The '''[[Kubernetes/Clusters]]''' page contains the definitive list of currently maintained clusters in the production realm, along with information about who manages them and each cluster's specific purpose. |
| | |
| | For information relating to the Kubernetes clusters in the cloud services realm, please see [[Kubernetes#Toolforge info|Toolforge info]]. |
|
| |
|
| == Packages == | | == Packages == |
| Line 11: |
Line 23: |
| == Services == | | == Services == |
|
| |
|
| A service in Kubernetes is an 'abstract way to expose an application running on a set of workloads as a network service'. | | A service in Kubernetes is an ''"abstract way to expose an application running on a set of workloads as a network service"''. That creates an overload of the term, as we also use the term ''"services"'' to describe how our various in-house developed applications are exposed to the rest of the infrastructure or the public. It's worthwhile to make sure one is on the same page as the other side when having a conversation around ''"services".'' Below there are some links to basic documentation about both concepts to help differentiate between them. |
|
| |
|
| * https://kubernetes.io/docs/concepts/services-networking/service/ | | * https://kubernetes.io/docs/concepts/services-networking/service/ |
| Line 17: |
Line 29: |
| * Learn more about [[Deployment pipeline/Migration/Tutorial | Migrating a service to kubernetes]] and [[Deployment pipeline]] generally. | | * Learn more about [[Deployment pipeline/Migration/Tutorial | Migrating a service to kubernetes]] and [[Deployment pipeline]] generally. |
|
| |
|
| == Debugging == | | == Deployment Charts == |
| | | We use a git repository called [[gerrit:plugins/gitiles/operations/deployment-charts/+/refs/heads/master/|operations/deployment-charts]] to manage all of the applications and deployments to Kubernetes clusters in the production realm. |
| For a quick intro into the debugging actions one can take during a problem in production look at [[Kubernetes/Helm]]. There will also be a guide posted under [[Kubernetes/Kubectl]]
| |
|
| |
|
| == Administration ==
| | See [[Kubernetes/Deployment Charts]] for more detailed information about the respository structure and its various functions. |
| === Add a new service ===
| |
| To add a new service to the clusters:
| |
|
| |
|
| * Ensure the service has it's ports registered at: [[Service ports]]
| | It primarly contains [[Helm]] charts and [[Helmfile]] deployments. |
| *Create deployment user/tokens in the puppet private (you can use a random generated password, no strict guideline for it) and public repos.
| |
| **Example 1
| |
| *** https://gerrit.wikimedia.org/r/c/labs/private/+/613101 (plus actual data in the private repo, see <code>1edf14c0</code> )
| |
| *** https://gerrit.wikimedia.org/r/c/operations/puppet/+/613104
| |
| **Example 2 - eventstreams-internal (T269160)
| |
| ***https://gerrit.wikimedia.org/r/655879 (plus the actual data in the private repo, see <code>6689496a</code> and <code>376c92ad</code>)
| |
| ***https://gerrit.wikimedia.org/r/c/operations/puppet/+/656129
| |
| * Add a Kubernetes namespace:
| |
| **Example 1
| |
| *** https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/645376 (ignore the change to calico/default-kubernetes-policy)
| |
| **Example 2 - eventstreams-internal (T269160)
| |
| ***https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/644612
| |
| *At this point, you can safely merge the change (after somebody from Service Ops validates it of course). Please do it though when you have time to run the following command, to avoid impacting other people rolling out changes later on.
| |
| *The first thing to do is to work in staging, updating the admin config.
| |
| **On deploy1002: <code>sudo -i; cd /srv/deployment-charts/helmfile.d/admin/staging/; kube_env admin staging; ./cluster-helmfile.sh -i apply</code>
| |
| **The command above should show you a change in namespaces/quotas/etc.. related to your new service. If this is not the case (for example, you also see other changes) ping somebody from the Service Ops team! There might be some work waiting to be applied.
| |
| *Then you can proceed to deploy the new service to staging for real. Don't worry for TLS (if needed) since in staging it will be added a default config for your service auto-magically. Different thing is Production, but there is a step later on about it :D
| |
| **On deploy1002: <code>cd /srv/deployment-charts/helmfile.d/services/YOUR-SERVICE-NAME-HERE; helmfile -e staging -i apply</code>
| |
| **The magic command above will show a diff related to the new service, make sure that everything looks fine and then hit Yes to proceed.
| |
| **You should now be able to test your new service in staging! You can use the handy endpoint <code>http(s)://staging.svc.eqiad.wmnet:$YOUR-SERVICE-PORT</code> to quickly test if everything works as expected.
| |
| *Now you can move to Production!
| |
| *Create certificates for the new service, if it has an HTTPS endpoint (remember that this step for staging is automatically handled for you, but for production it is not).
| |
| **[[Enable TLS for Kubernetes deployments]]
| |
| *If the new service requires specific secrets, commit them to <code>/srv/private/hieradata/role/common/deployment_server.yaml</code>
| |
| *At this point, you need to update the admin config for eqiad and codfw (if you have configs for both of course):
| |
| **On deploy1002: <code>sudo -i; cd /srv/deployment-charts/helmfile.d/admin/codfw/; kube_env admin codfw; ./cluster-helmfile.sh -i apply</code>
| |
| **On deploy1002: <code>sudo -i; cd /srv/deployment-charts/helmfile.d/admin/eqiad/; kube_env admin eqiad; ./cluster-helmfile.sh -i apply</code>
| |
| *Then the final step, namely deploying the new service:
| |
| **On deploy1002: <code>cd /srv/deployment-charts/helmfile.d/services/YOUR-SERVICE-NAME-HERE; helmfile -e codfw -i apply</code>
| |
| **On deploy1002: <code>cd /srv/deployment-charts/helmfile.d/services/YOUR-SERVICE-NAME-HERE; helmfile -e eqiad -i apply</code>
| |
| The service can now be accessed via the registered port on any of the kubernetes nodes (for manual testing).
| |
|
| |
|
| If you need the service to be easily accessible from outside of the cluster, you might want to add [[LVS#Add%20a%20new%20load%20balanced%20service|Add a new load balanced service]].
| | The services and deployments that are defined within the repository are a combination of: |
|
| |
|
| === Rebooting a worker node ===
| | * WMF software, running on [[Kubernetes/Images#Services%20images|service images]] managed by the [[deployment pipeline]] |
| | * WMF forks of third-party software, also running on [[Kubernetes/Images#Services images|service images]] managed by the [[deployment pipeline]] |
| | * WMF builds of third-party software, running on [[Kubernetes/Images#Production images|production images]] and built with [https://doc.wikimedia.org/docker-pkg/ docker-pkg] |
| | See [[Kubernetes/Deployments]] for instructions regarding day-to-day deployment of Kubernetes [[Kubernetes#services|services]]. |
|
| |
|
| ==== The unpolite way ==== | | == Debugging == |
| To reboot a worker node, you can just reboot it in our environment. The platform will understand the event and respawn the pods on other nodes. However the system does not automatically rebalance itself currently (pods are not rescheduled on the node after it has been rebooted)
| |
|
| |
|
| ==== The polite way (recommended) ====
| | For a quick intro into the debugging actions one can take during a problem in production look at [[Kubernetes/Helm]]. There will also be a guide posted under [[Kubernetes/Kubectl]] |
|
| |
|
| If you feel like being more polite, use kubectl drain, it will configure the worker node to no longer create new pods and move the existing pods to other workers. Draining the node will take time. Rough numbers on 2019-12-11 are at around 60 seconds.
| | == Administration == |
| | | See [[Kubernetes/Administration]] for collected instructions and runbooks for such tasks as: |
| <syntaxhighlight lang="shell-session">
| |
| # kubectl drain kubernetes1001.eqiad.wmnet
| |
| # kubectl describe pods --all-namespaces | awk '$1=="Node:" {print $NF}' | sort -u
| |
| kubernetes1002.eqiad.wmnet/10.64.16.75
| |
| kubernetes1003.eqiad.wmnet/10.64.32.23
| |
| kubernetes1004.eqiad.wmnet/10.64.48.52
| |
| kubernetes1005.eqiad.wmnet/10.64.0.145
| |
| kubernetes1006.eqiad.wmnet/10.64.32.18
| |
| # kubectl get nodes
| |
| NAME STATUS ROLES AGE VERSION
| |
| kubernetes1001.eqiad.wmnet Ready,SchedulingDisabled <none> 2y352d v1.12.9
| |
| kubernetes1002.eqiad.wmnet Ready <none> 2y352d v1.12.9
| |
| kubernetes1003.eqiad.wmnet Ready <none> 2y352d v1.12.9
| |
| kubernetes1004.eqiad.wmnet Ready <none> 559d v1.12.9
| |
| kubernetes1005.eqiad.wmnet Ready <none> 231d v1.12.9
| |
| kubernetes1006.eqiad.wmnet Ready <none> 231d v1.12.9
| |
| </syntaxhighlight>
| |
| | |
| When the node has been rebooted, it can be configured to reaccept pods using '''kubectl uncordon''', e.g.
| |
| <syntaxhighlight lang="shell-session">
| |
| # kubectl uncordon kubernetes1001.eqiad.wmnet
| |
| # kubectl get nodes
| |
| NAME STATUS ROLES AGE VERSION
| |
| kubernetes1001.eqiad.wmnet Ready <none> 2y352d v1.12.9
| |
| kubernetes1002.eqiad.wmnet Ready <none> 2y352d v1.12.9
| |
| kubernetes1003.eqiad.wmnet Ready <none> 2y352d v1.12.9
| |
| kubernetes1004.eqiad.wmnet Ready <none> 559d v1.12.9
| |
| kubernetes1005.eqiad.wmnet Ready <none> 231d v1.12.9
| |
| kubernetes1006.eqiad.wmnet Ready <none> 231d v1.12.9
| |
| </syntaxhighlight>
| |
| | |
| The pods are not rebalanced automatically, i.e. the rebooted node is free of pods initially.
| |
| | |
| === Restarting calico-node ===
| |
| | |
| calico-node maintains a BGP session with the core routers if you intend to restart this service you should use the following procedure
| |
| | |
| # drain the node on the kube controler as shown above
| |
| # <code>systemctl restart calico-node</code> on the kube worker
| |
| # Wait for BGP sessions on the coure router to re-established
| |
| # uncordon the node on the kube controler as shown above
| |
| | |
| you can use the following command on the cour routers to check BGP status (use <code>match 64602</code> for codfw)
| |
| | |
| <syntaxhighlight lang="shell-session">
| |
| # show bgp summary | match 64601
| |
| 10.64.0.121 64601 220 208 0 2 32:13 Establ
| |
| 10.64.0.145 64601 824512 795240 0 1 12w1d 21:45:51 Establ
| |
| 10.64.16.75 64601 161 152 0 2 23:25 Establ
| |
| 10.64.32.18 64601 824596 795247 0 2 12w1d 21:46:45 Establ
| |
| 10.64.32.23 64601 130 123 0 2 18:59 Establ
| |
| 10.64.48.52 64601 782006 754152 0 3 11w4d 11:13:52 Establ
| |
| 2620:0:861:101:10:64:0:121 64601 217 208 0 2 32:12 Establ
| |
| 2620:0:861:101:10:64:0:145 64601 824472 795240 0 1 12w1d 21:45:51 Establ
| |
| 2620:0:861:102:10:64:16:75 64601 160 152 0 2 23:25 Establ
| |
| 2620:0:861:103:10:64:32:18 64601 824527 795246 0 1 12w1d 21:46:45 Establ
| |
| 2620:0:861:103:10:64:32:23 64601 130 123 0 2 18:59 Establ
| |
| 2620:0:861:107:10:64:48:52 64601 782077 754154 0 2 11w4d 11:14:13 Establ
| |
| </syntaxhighlight>
| |
| | |
| === Restarting specific components ===
| |
| | |
| kube-controller-manager and kube-scheduler are components of the API server. In production multiple ones run and perform via the API an election to determine which one is the master. Restarting both is without grave consequences so it's safe to do. However both are critical components in as such that there are required for the overall cluster to function smoothly. kube-scheduler is crucial for node failovers, pod evictions, etc while kube-controller-manager packs multiple controller components and is critical for responding to pod failures, depools etc.
| |
| | |
| commands would be <syntaxhighlight lang="bash">
| |
| sudo systemctl restart kube-controller-manager
| |
| sudo systemctl restart kube-scheduler
| |
| </syntaxhighlight>
| |
| | |
| === Restarting the API server ===
| |
| | |
| It's behind LVS in production, it's fine to restart it as long as enough time is given between the restarts across the cluster.
| |
| <syntaxhighlight lang="bash">
| |
| sudo systemctl restart kube-apiserver
| |
| </syntaxhighlight>
| |
| | |
| ===Reinitialize a complete cluster===
| |
| If, for whatever reason, we need to reinitialize a kubernetes cluster on a new etcd backing store. The following steps could be used as guideline. They might also help in understanding how the clusters are set up and how to set up new ones.
| |
| | |
| #Create puppet change, pointing k8s (and calico) to the new etcd cluster, see:
| |
| ##{{Gerrit|558355}} and {{Gerrit|558473}}
| |
| #Populate IPPool and BGP nodes in the new calico etcd backend
| |
| ##On a random node of the kubernetes cluster:<syntaxhighlight lang="bash">
| |
| cp /etc/calico/calicoctl.cfg .
| |
| # Modify the etcdEndpoints config in ./calicoctl.cfg to point to new etcd
| |
| # Set asNumber (64601 for eqiad, 64603 for codfw)
| |
| calicoctl config set asNumber 6460X --config=calicoctl.cfg
| |
| calicoctl config set nodeToNodeMesh off --config=calicoctl.cfg
| |
| # FIXME: This assumes we still have access to the old etcd to read bgppeer
| |
| # and ippool data from.
| |
| calicoctl get -o yaml bgppeer | calicoctl create -f - --config=calicoctl.cfg
| |
| calicoctl get -o yaml ippool | calicoctl create -f - --config=calicoctl.cfg
| |
|
| |
|
| # Create a basic default profile for the kube-system namespace in order to
| | * [[Kubernetes/Administration#Rebooting worker nodes|Rebooting worker nodes]] |
| # allow kube-system tiller to talk to the kubernetes API to deploy the
| | * [[Kubernetes/Administration#Restarting%20specific%20components|Restarting specific components]] |
| # calico-policy-controller (avoid catch-22).
| | * [[Kubernetes/Administration#Managing pods, jobs and cronjobs|Managing pods, jobs, and, cronjobs]] |
| #
| |
| # When the calico-policy-controller is started, it will sync things and this
| |
| # simple profile will be updated and set up correctly.
| |
| calicoctl create -f - --config=calicoctl.cfg <<_EOF_
| |
| - apiVersion: v1
| |
| kind: profile
| |
| metadata:
| |
| name: k8s_ns.kube-system
| |
| tags:
| |
| - k8s_ns.kube-system
| |
| spec:
| |
| egress:
| |
| - action: allow
| |
| destination: {}
| |
| source: {}
| |
| ingress:
| |
| - action: allow
| |
| destination: {}
| |
| source: {}
| |
| _EOF_
| |
| </syntaxhighlight>
| |
| #Schedule downtime for | |
| ##services running on the cluster
| |
| ##kubernetes nodes and master<syntaxhighlight lang="bash">
| |
| sudo cookbook sre.hosts.downtime -r 'Reinitialize eqiad k8s cluster with new etcd' -t TXXX -H 4 'A:eqiad and (A:kubernetes-masters or A:kubernetes-workers)'
| |
| </syntaxhighlight>
| |
| #Depool services from discovery/edge caches
| |
| #Delete all helmfile managed namespaces (to be sure we see errors/missing things early)
| |
| #Disable puppet on master and k8s nodes<syntaxhighlight lang="bash">
| |
| sudo cumin 'A:eqiad and (A:kubernetes-masters or A:kubernetes-workers)' "disable-puppet 'Reinitialize eqiad k8s cluster with new etcd - TXXXX'"
| |
| </syntaxhighlight>
| |
| #Stop apiserver and calico node on k8s nodes | |
| #Merge puppet changes
| |
| #Enable and run puppet on the k8s nodes
| |
| #Enable puppet on 1 apiserver and run it
| |
| #Disable puppet on apiserver again
| |
| #Edit <code>/etc/default/kube-apiserver</code> to disable PodSecurityPolicy controller
| |
| #Start API server (running without PodSecurityPolicy controller now) | |
| #Run <code>deployment-chars/helmfile.d/admin/initialize_cluster.sh</code> for the cluster
| |
| #Restart kubelet on all kubernetes nodes<syntaxhighlight lang="bash">
| |
| sudo cumin 'A:eqiad and A:kubernetes-workers' 'systemctl restart kubelet'
| |
| </syntaxhighlight>
| |
| #Enable puppet on kubernetes masters again and run it. This will restart API server with PodSecurityPolicy controller
| |
| #Run <code>helmfile.d/admin/eqiad/cluster-helmfile.sh</code>
| |
| #Deploy all services via a for loop and helmfile sync commands
| |
|
| |
|
| === Switch the active staging cluster (eqiad<->codfw) === | | === See also === |
| We do have one staging cluster per DC, mostly to separate staging of kubernetes and components from staging of the services running on top of it. To provide staging services during work on one of the clusters, we can (manually) switch between the DCs:
| |
|
| |
|
| * Switch staging.svc.eqiad.wmnet to point to the new active k8s cluster (we should have a better solution/DNS name for this at some point) | | * [[Kubernetes/Clusters/New|Adding a new Kubernetes cluster]] |
| ** https://gerrit.wikimedia.org/r/c/operations/dns/+/667982 | | * [[Kubernetes/Deployments|Deployments on Kubernetes]] |
| * Switch the definition of "staging" on the deployment servers:
| | * [[Kubernetes/Kubernetes Education|Kubernetes Education]] |
| ** https://gerrit.wikimedia.org/r/c/operations/puppet/+/667996
| |
| * Switch CI and releases to the other kubestagemaster:
| |
| ** https://gerrit.wikimedia.org/r/c/operations/puppet/+/668114 | |
|
| |
|
| ==See also== | | ==Toolforge Info== |
| *[[Portal:Toolforge/Admin/Kubernetes|Toolforge Kubernetes cluster design and administration]] | | *[[Portal:Toolforge/Admin/Kubernetes|Toolforge Kubernetes cluster design and administration]] |
| *[[Help:Toolforge/Web|Toolforge Kubernetes webservice help]] | | *[[Help:Toolforge/Web|Toolforge Kubernetes webservice help]] |
| Line 227: |
Line 66: |
|
| |
|
| [[Category:Kubernetes]] | | [[Category:Kubernetes]] |
| | [[Category:SRE Service Operations]] |
