You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

PAWS/Tools/Admin/Chico's notes

From Wikitech-static
< PAWS‎ | Tools‎ | Admin
Revision as of 20:44, 1 March 2018 by imported>Chico Venancio (Persisting https://etherpad.wikimedia.org/p/PAWS-beta for now, should merge with Admin)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Notes on setting up a PAWS staging env in toolsbeta VPS project (T188428)

Using https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ as docs

  • Doesn't seem like PAWS is properly puppetized
    • I see the apt-pinning defnitions, but not where tools-paws-master-01 installs needed packages (k8s, docker, etc)
  • Docker and k8s repos are defined for Xenial, though tools-paws-master-01 is stretch
    • Docker does have current stretch versions, k8s does not (left docker as strech, k8s as xenial)
  • Unsure about cgroup driver used by docker. Official docs says to place { "exec-opts": ["native.cgroupdriver=systemd"] } in /etc/docker/daemon.json since the prod version does not have that I'm ommiting it for now
  • swap needs to be turned off for docker, done manually
  • started the k8s cluster with flannel
    • kubeadm init --pod-network-cidr=10.244.0.0/16
  • Allow user chicocvenancio to use k8s
    • chicocvenancio@toolsbeta-paws-master-01:~$ mkdir -p $HOME/.kube
    • chicocvenancio@toolsbeta-paws-master-01:~$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    • chicocvenancio@toolsbeta-paws-master-01:~$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
  • Docs say to set /proc/sys/net/bridge/bridge-nf-call-iptables to 1, it already was 1
  • get flannel pods (docs mentions v0.9.1 I used latest version)
  • exported Yuvi's git-crypt key
    • yuvipanda@tools-paws-master-01:~/paws$ sudo -E git-crypt export-key /tmp/paws-key
    • chicocvenancio@tools-paws-master-01:~$ sudo chown chicocvenancio /tmp/paws-key
    • chicocvenancio@tools-paws-master-01:~$ scp /tmp/paws-key toolsbeta-paws-master-01.toolsbeta:~/paws-key
  • Intalled git-crypt
    • chicocvenancio@toolsbeta-paws-master-01:~$ sudo apt-get install git-crypt
  • ran into Error: could not find tiller
    • fixed with `helm init`
  • Tiller won't start due to a lack of nodes
  • Create new node
    • Is there a way to adhere to naming convention when using instance count in horizon?
    • toolsbeta-paws-worker-1001
      • Since its not puppetized, going for manual again
    • Node joining brings up tiller
  • Once tiller is up we can run "sudo ./build.py deploy prod --install" to install PAWS
    • We need to fix two things Yuvi did CLI and did not push to repo (yuvi's .bash_history invaluable to get these right)
      • Tiller RBAC
        • Done now by setting a non ideal permissive clusterrolebinding
          • chicocvenancio@toolsbeta-paws-master-01:~$ kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --user=admin --user=kubelet --group=system:serviceaccounts
      • the hub_db pvc is not defined at all in the repo, found the definition in yuvi's .bash_history
        • chicocvenancio@toolsbeta-paws-master-01:~$ kubectl -n prod apply -f /mnt/nfs/labstore-secondary-home/yuvipanda/paw-c/hub-pv.yaml
  • TODO: setup a new OAuth consumer for PAWS-beta
  • TODO: stop these annoying pre-puller deamonsets
    • This is actually not annoying and good once I pointed them to working docker repositories
  • Right now paws-beta uses a completely different way (WMCS-wise) for traffic ingress, I thought this simpler than copying the paws-proxy instance, in fact we can probably drop those instances (VPS cloud project) and improve production after some testing
    • I did not get the ideal k8s LoadBalancer service to work with external IPs, instead I used a NodePort service and pointed a webproxy to one of the nodes (any will do)
      • This does mean that if that node fails the site will proxy will fail, which is NOT ok for production
  • Differences between PAWS-beta and prod:
    • Already without the query-killer image
    • Deploy-hook image uses artful and not zesty
    • Per above, traffic ingress is different