User:Accraze/MachineLearning/Local Kserve

Summary

This page is a guide for installing the KServe stack locally using WMF tools and images. The install steps diverge from the official KServe quick_install script in order to run on WMF infrastructure. All upstream changes to YAML configs were first published in the KServe chart's README for the deployment-charts repository. In deployment-charts/custom_deploy.d/istio/ml-serve there is the config.yaml that we apply in prod.

Minikube

We are running a small cluster using Minikube, which can be installed with the following command:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

To match production, we want to make sure we set our k8s version to v.1.16.15:

minikube start --kubernetes-version=v1.16.15 --insecure-registry="docker-registry.wikimedia.org:443"

You will also need to install kubectl, or you can use the one provided by minikube with an alias:

alias kubectl="minikube kubectl --"

Istio

Istio is installed using the istioctl package, which has been added to the WMF APT repository, you can use it (https://wikitech.wikimedia.org/wiki/APT_repository, debian buster). See: https://apt-browser.toolforge.org/buster-wikimedia/main/ , we want to install Istio 1.9.5 (istioctl: 1.9.5-1)

For Wikimedia servers and Cloud VPS instances, the repositories are automatically configured via Puppet. You can install it as follows

sudo apt install istioctl

Now we need to create the istio-system namespace:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: istio-system
  labels:
    istio-injection: disabled
EOF

Next you will need to create a file called istio-minimal-operator.yaml:

apiVersion: install.istio.io/v1beta1
kind: IstioOperator
spec:
  values:
    global:
      proxy:
        autoInject: disabled
      useMCP: false
      # The third-party-jwt is not enabled on all k8s.
      # See: https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens
      jwtPolicy: first-party-jwt

  meshConfig:
    accessLogFile: /dev/stdout

  addonComponents:
    pilot:
      enabled: true

  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true

Next, you can apply the manifest using istioctl:

/usr/bin/istioctl-1.9.5 manifest apply -f istio-minimal-operator.yaml -y

Knative

We are currently running Knative Serving v0.18.1. First let's install the Knative serving-crds.yaml. The CRDs are copied from upstream: https://github.com/knative/serving/releases/download/v0.18.1/serving-crds.yaml

We have them included in our deployment-charts repo: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/charts/knative-serving-crds/templates/crds.yaml

You can install using the following command:

kubectl apply -f crds.yaml

For knative gateway there is a specific setting in a config-map to change, otherwise it won't work.

Next we need to add registries skipping tag resolving etc.:

kubectl edit configmap config-deployment -n knative-serving

Add the following config in data:

...
registriesSkippingTagResolving: "kind.local,ko.local,dev.local"

Images

• Webhook: https://docker-registry.wikimedia.org/knative-serving-webhook/tags/
• Queue: https://docker-registry.wikimedia.org/knative-serving-queue/tags/
• Controller: https://docker-registry.wikimedia.org/knative-serving-controller/tags/
• Autoscaler: https://docker-registry.wikimedia.org/knative-serving-autoscaler/tags/
• Activator: https://docker-registry.wikimedia.org/knative-serving-activator/tags/
• Net-istio webhook: https://docker-registry.wikimedia.org/knative-net-istio-webhook/tags/
• Net-istio controller: https://docker-registry.wikimedia.org/knative-net-istio-controller/tags/

KServe

Images

• KServe agent: https://docker-registry.wikimedia.org/kserve-agent/tags/
• Kserve controller: https://docker-registry.wikimedia.org/kserve-controller/tags/
• KServe storage-initializer: https://docker-registry.wikimedia.org/kserve-storage-initializer/tags/