You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Kubernetes/Clusters/Add or remove nodes: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Volans
m (→‎Add node to BGP: Fix link, take 2)
imported>JMeybohm
 
Line 53: Line 53:
Then use the [[Server_Lifecycle/Reimage#How_to_run_it|re-image script]] to image you nodes, apply puppet and so on.
Then use the [[Server_Lifecycle/Reimage#How_to_run_it|re-image script]] to image you nodes, apply puppet and so on.


{{warning| Reimaging will bring the node up with Kernel 4.9 and Kernel 4.19 will be installed by puppet later. So a reboot after the initial puppet run is required! }}
{{warning| Reimaging will make the node join the cluster automatically. But for them to be fully functional, we need a puppet run on docker-registry nodes, see: {{PhabT|T273521}}. Try to avoid joining nodes in deployment windows. }}
 
Make sure to run puppet on [[Docker-registry]] nodes soon after the reimage:
<syntaxhighlight lang="bash">
sudo cumin A:docker-registry 'run-puppet-agent -q'
</syntaxhighlight>


== Add to conftool/LVS ==
== Add to conftool/LVS ==

Latest revision as of 18:07, 17 March 2022

Intro

This is a guide for adding or removing nodes from existing Kubernetes clusters.


Adding a node

With the creation of a Kubernetes cluster a Puppet role for the workers has been created (see: Kubernetes/Clusters/New#General_Puppet/hiera_setup)

Add node specific hiera data

You need to add node specific data, like the failure-domain/topology annotations:

This can be done by creating the file hieradata/hosts/foo-node1001.yaml:

profile::kubernetes::node::kubelet_node_labels:
  - failure-domain.beta.kubernetes.io/region=codfw
  - failure-domain.beta.kubernetes.io/zone=row-c

You can get the right region and zone values for your node from Netbox, like https://netbox.wikimedia.org/search/?q=foo-node1001

Add node to BGP

Add to homer

Nodes (in the calico setup) need to be able to establish BGP with the core routers. To be able to, they need to be added to as neighbors in config/sites.yaml of the operations/homer/public repository:

eqiad:
  [...]
  foo_neighbors:
    foo_node1001: {4: <Node IPv4>, 6: <Node IPv6}

You will have to run homer, once that change is merged. See: Homer#Running Homer from cluster management hosts (recommended)

Add to calico

In addition, all nodes are BGP peers for each other. So we need to extend the the hiera key profile::calico::kubernetes::bgp_peers for this Kubernetes cluster with the new nodes FQDN in: hieradata/role/<DATACENTER>/<CLUSTER>/worker.yaml e.g.:

profile::calico::kubernetes::bgp_peers: 
- cr1-eqiad.wikimedia.org 
- cr2-eqiad.wikimedia.org
[...]
- foo_node1001.eqiad.wmnet

Reimage the node

Then use the re-image script to image you nodes, apply puppet and so on.

Make sure to run puppet on Docker-registry nodes soon after the reimage:

sudo cumin A:docker-registry 'run-puppet-agent -q'

Add to conftool/LVS

If the Kubernetes cluster is exposing services via LVS (production clusters usually do, staging ones don't), you need to add the nodes FQDN to the cluster in conftool-data as well. For eqiad in conftool-data/node/eqiad.yaml like:

eqiad:
  foo:
    [...]
    foo_node1001.eqiad.wmnet: [kubesvc]

Done

Please ensure you've followed all necessary steps from Server_Lifecycle#Staged_->_Active

Your node should now join the cluster and have workload scheduled automatically (like calico daemonsets). You can check with:

kubectl get nodes

Removing a node

Drain workload

First step to remove a node is to drain workload from it. This is also to ensure that the workload actually still fits the cluster:

kubectl drain --ignore-daemonsets foo-node1001.datacenter.wmnet

However, some workloads might be using volumes mounted to local storage and to drain those you need to add a second option:

kubectl drain --ignore-daemonsets --delete-local-data foo-node1.datacenter.wmnet

You can verify success by looking at what is still scheduled on the node:

kubectl describe node foo-node1001.datacenter.wmnet

Decommission

You can now follow the steps outlined in Server_Lifecycle#Active_->_Decommissioned

Ensure to also remove:

Delete the node from Kubernetes API

The step left is to delete the node from Kubernetes:

kubectl delete node foo-node1001.datacenter.wmnet