SLO

Service Level Objective (SLO) and Service Level Indicators (SLI)

Rationale: We’d love it if all our systems responded instantly and worked 100% of the time, but we also know that’s unrealistic. By choosing specific objectives based on what is important to our users, we can aim to keep our users happy, and still be able to prioritize other work as long as we’re meeting those objectives. If the performance starts to dip down toward the threshold, objectively we know it’s time to refocus on short-term reliability work and put other things on hold. And by breaking up our complex production landscape into individual services, each with its own SLOs, we know where to focus that work.

Service Level Indicator

a measurement of a behavior of a system that can be used to monitor the system's function. In the Service Level context ideally expressed as percentage

Examples

• Speed of Response: requests that are handled under a set threshold - "requests that are fulfilled under 500 ms divided by all requests * 100"
• Success: requests that are handled error free - "all requests but return code 500 divided by all requests *100"
• Freshness: pages that are served updated - "pages served are outdated less than 5 seconds/all pages served * 100"

Service Level Objective

Once we have SLIs we can reason about an objective

Examples

• We want 99% of all request to be faster than 500ms
• No more than 0.1% of errors
• At most 1% of pages are served outdated

Published SLOs

• etcd main cluster
• API Gateway
• docker-registry
• Varnish caching
• Logstash

SLO reporting

Quarterly, but offset by one month, i.e. December, January, February.

Example:

slide of Tuning Sessions for SLOs

External links

• Introduction to SLOs at the Wikimedia Foundation - Google Slide Link
• Implementing Service Level Objectives - O'Reilly book - Sample Chapters
• Service Level Objective Intro in Site Reliability Engineering (SRE) book by Google
• Dependency Math at Google
• [1] Meaningful availability - downtime multiplied with the number of users online
• SLO Workshop by Google at SRECon 2018

Service Level Objectives at the foundation

• Worksheet template

Additional Keywords: Error Budget, RED Method, four golden signals