Machine Learning/LiftWing/Inference Services
Our Machine Learning models are hosted as Inference Services (isvc), which are Custom Resource Definitions (CRDs), an extension of the Kubernetes API. The isvc CRD is provided by KServe, which utilizes Knative and Istio and provides serverless, asynchronous micro-services designed for performing inference. These services are written in Python and use the Tornado framework.
You can download the repo (with commit-msg hook) from Gerrit with the following command:
git clone "https://gerrit.wikimedia.org/r/machinelearning/liftwing/inference-services" && (cd "inference-services" && mkdir -p .git/hooks && curl -Lo `git rev-parse --git-dir`/hooks/commit-msg https://gerrit.wikimedia.org/r/tools/hooks/commit-msg; chmod +x `git rev-parse --git-dir`/hooks/commit-msg)
We use the ML-Sandbox to develop and test isvc code in a KServe environment.
Examples on how to use the ML-Sandbox as a development environment: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/ML-Sandbox/Usage-Examples
Each Inference Service is a k8s pod that can be comprised of different containers (transformer, predictor, explainer, storage-initializer). When we are ready to deploy a service, we first need to create a production image for each service and publish it to the WMF Docker Registry using the Deployment Pipeline.
- Production Image Development Guide: Machine Learning/LiftWing/Inference Services/Production Image Development
- articlequality: https://docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-articlequality/tags/
- draftquality: https://docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-draftquality/tags/
- editquality: https://docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-editquality/tags/
- topic: https://docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-topic/tags/
Since the inference service code is stored in a monorepo, we manage all individual Inference Service images using separate test and publish pipelines on Jenkins.
All pipelines are configured in the
.pipeline/config.yaml file in the project root and use PipelineLib to define the different stages in each pipeline. Once you have created a BlubberFile and configured a pipeline, you will need to add them into the Deployment Pipeline. This will require you to define the jobs and set triggers in the jenkins job builder spec in the integrations/config repo.
- Integrating New Pipelines guide: https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Migration/Tutorial#project-pipelines.yaml
Currently, our test/build pipelines are triggered whenever we edit code for a given InferenceService. When we push a new CR to Gerrit, jenkins-bot starts a job on the isvc's pipeline. This job uses the tox tool to run a test suite on our isvc code: right now it's just doing flake8 and then running the black formatter, but could be expanded for different model types. If the code passes the checks, then we attempt to build the full production image (as defined in the blubberfile).
The publish pipelines are run as post-merge jobs. Whenever a CR is merged on Gerrit, the post-merge jobs will run (as seen in Zuul) and will attempt to re-build the production image again and, if successful, will be published to the WMF Docker Registry. After the image has been pushed, PipelineBot will respond with a message on the Gerrit CR with the newly tagged image uri.
Each of our pipelines run jobs on Jenkins and are managed via Zuul:
- editquality: https://integration.wikimedia.org/ci/job/inference-services-pipeline-editquality/
- editquality-publish: https://integration.wikimedia.org/ci/job/inference-services-pipeline-editquality-publish/
- articlequality: https://integration.wikimedia.org/ci/job/inference-services-pipeline-articlequality/
- articlequality-publish: https://integration.wikimedia.org/ci/job/inference-services-pipeline-articlequality-publish/
- draftquality: https://integration.wikimedia.org/ci/job/inference-services-pipeline-draftquality/
- draftquality-publish: https://integration.wikimedia.org/ci/job/inference-services-pipeline-draftquality-publish/
- topic: https://integration.wikimedia.org/ci/job/inference-services-pipeline-topic/
- topic-publish: https://integration.wikimedia.org/ci/job/inference-services-pipeline-topic-publish/
- outlink: https://integration.wikimedia.org/ci/job/inference-services-pipeline-outlink/
- outlink-publish: https://integration.wikimedia.org/ci/job/inference-services-pipeline-outlink-publish/
- outlink-transformer: https://integration.wikimedia.org/ci/job/inference-services-pipeline-outlink-transformer/
- outlink-transformer-publish: https://integration.wikimedia.org/ci/job/inference-services-pipeline-outlink-transformer-publish/