Service level objective

What it is

A service level objective is a target for how well a service
behaves, written so precisely that anyone can check whether it was
met: a share of requests that must succeed, or must be answered fast
enough, over a stated period. Almost everyone shortens it to SLO.
Google's Site Reliability Engineering book defines it as a target
value, or a range of values, for one level of service, as measured
by an indicator.

That indicator is the service level indicator, or SLI: a quantity
defined with care to describe one aspect of the service users
actually get. Request latency is the usual first choice, with error
rate and throughput close behind. The SRE Workbook recommends
writing every SLI as a ratio, good events divided by all events, so
that it runs from 0%, where nothing works, to 100%, where nothing is
broken. The objective is then a line drawn on that scale: the
indicator must stay on the right side of a target, or between a
lower and an upper bound.

A complete objective names everything needed to check it. The book's
example reads: 99% of Get RPC calls, averaged over one minute,
complete in under 100 ms, measured across all backend servers. One
indicator can carry several targets at once, such as 90% of calls
under 1 ms, 99% under 10 ms and 99.9% under 100 ms, because one
percentile rarely describes a whole distribution. The Workbook also
separates the specification, the outcome that matters to users, from
the implementation, the place it is measured: a server log, a prober
running a browser, or code in the page itself. Each one sees
different failures, and the choice belongs in the definition.

An SLO is not an SLA. A service level agreement is a contract with
users, explicit or implicit, that attaches consequences to meeting
or missing the objectives inside it, most visibly a rebate or a
penalty. The book's test is to ask what happens when the objective
is missed: if nothing explicit follows, it is an SLO. A footnote
adds that people who talk about a violated SLA nearly always mean a
missed SLO, since a real breach of an SLA can end in court for
breach of contract. Google Search, the book notes, has no SLA for
the public, but an outage of it still costs reputation and
advertising revenue.

Why it matters

Without a stated objective, users invent their own. The SRE book
describes both ways that goes wrong: people who trust a service more
than they should, and prospective users who think it flakier than it
is and stay away. Its example is Chubby, Google's lock service for
distributed systems. Global outages of it were so rare that other
services quietly came to depend on it never failing, and those
services broke on the rare occasions it did. The answer was to make
the global instance meet its SLO without exceeding it by much: in
any quarter where real failures had not pushed availability below
the target, SRE took the system down on purpose.

The objective also settles the argument about how reliable is
reliable enough. "100% reliability is the wrong target", writes the
SRE Workbook, and its reasons are practical. Components fail at the
same time however they are built. The chain of networks and devices
between a service and its customers fails too, so the customers
would never see the 100%. Each extra nine costs more than the last
while its value to users approaches zero. And since change is the
leading cause of outages, a service that may never fail can never be
updated either. Once the target is below 100%, the gap between the
target and perfection is room to spend on releases, experiments and
planned maintenance: the error budget, which is simply 100% minus
the SLO. Someone has to own that trade between shipping features and
staying up; the Workbook names the CTO in a small organisation and
the product owner or product manager in a larger one.

Finally, the objective turns monitoring into decisions. The book
describes a loop: measure the indicators, compare them with the
objectives, decide whether anything needs doing, and do it. If
latency is climbing and will cross the line within a few hours, the
team checks whether the servers are CPU-bound and adds more of them.
Without the objective, the book points out, nobody can say whether
to act, or when.

What it looks like in practice

Take a checkout API and an objective written the way the Workbook
suggests, with the indicator separate from where it is measured, the
window stated and the reasoning kept beside it:

```text
Service:    checkout API
SLI:        share of POST /checkout requests answered without a 5xx
            status, counted in the server's request log
Objective:  99.9% over a rolling 28-day window
Rationale:  observational, rounded down from four weeks of data
Owner:      product manager, with the on-call engineers
Review:     monthly until it settles, then quarterly
```

The window matters as much as the percentage. The Workbook prefers a
rolling window to a calendar month, because users do not forget an
outage on the last day of a month when the next one begins, and a
whole number of weeks, so that every window holds the same number of
weekends; four weeks is its general-purpose choice. That window is
28 × 24 × 60 = 40,320 minutes, and 0.1% of it is a little over 40
minutes of complete outage. On a 30-day month the same

99.9% allows 43.2 minutes, the figure in the book's availability
table.

Time is the cruder way to count, though. The SRE book explains that
Google measures availability as the success rate of requests rather
than as uptime, because a service spread across the world is seldom
entirely up or entirely down. Counted in requests, the Workbook's
example runs: at 99.9%, 3 million requests over four weeks leave
room for 3,000 failures, and an outage that causes 1,500 of them
uses half the budget. Counting requests also prices the partial
failures that uptime misses. At steady traffic, 20 minutes in which
half of all requests fail costs as much as 10 minutes of total
outage, which is roughly a quarter of the 28-day budget.

In Prometheus, with a request counter labelled by status, the
indicator over the window is one ratio, modelled on the Workbook's
own example:

```promql
sum(rate(http_requests_total{job="checkout", status!~"5.."}[28d]))
  /
sum(rate(http_requests_total{job="checkout"}[28d]))
```

On this site, the service levels page sits on the agreement side of
the line. For a system the company runs on a client's behalf, the
response time for an incident, the hours covered, the availability
it is held to and the maintenance windows are written into the
agreement for that system, and they are set per system because a
nightly report and a payment service need different answers. The
first-party error monitoring and tracing built by the founder as
part of current work under NextMariner surfaces through a set of
operator views, one of them for service level objectives. And the
note on selling the last ticket exactly once turns a stock invariant
into one: check it on a schedule, chart it and page on any drift, so
that zero oversells is an objective with an alert behind it rather
than a hope.

Where it goes wrong

The target is copied from last month's dashboard. Taking whatever
the system achieves today feels safe, but the SRE book warns that a
target adopted without reflection can lock a team into heroic
efforts to keep meeting it, on a system that cannot improve without
a redesign. Work back from what users need, and write the reasoning
down beside the number: the Workbook asks for the rationale, and for
whether the figures came from experiment or observation, even when
they are frankly ad hoc. Review the document monthly at first and
quarterly once it settles.

The objective is 100%, or a string of nines nobody has priced. A
system that is always available, or scales without limit, is
expensive to build and run, and better than users would notice.
Start with a loose target and tighten it as you learn the system;
the book prefers that to a strict target that later has to be
relaxed because it proved out of reach.

An average stands in for what users feel. In the book's example a
typical request takes about 50 ms while 5% of requests take twenty
times as long, and a graph of the average shows nothing happening
all day. Write latency objectives as percentiles, or as a share of
requests under a threshold such as 300 ms, never as a mean.

There are more objectives than anyone can defend. A dozen SLOs per
service dilutes all of them. The book's test is blunt: if quoting an
objective can never win an argument about priorities, it is probably
not worth having. Keep the few that cover what users actually do.

Missing the objective changes nothing. The Workbook calls this an
SLO without teeth: compliance becomes one more KPI on a report
instead of a reason to decide anything. Agree in advance what
happens when the budget is spent, and have the product manager, the
developers and the SREs sign it. The Workbook's example error budget
policy pauses ordinary changes while the service is outside its
objective, allowing only the highest-priority bug fixes and security
work, and says in so many words that the policy is not a punishment.

The SLO is the same number as the SLA. When the internal objective
matches the contract, the first a team hears of trouble is the
breach. The book recommends a tighter internal objective than the
one advertised, as a buffer for chronic problems. The opposite
mistake matters too: a service that runs far better than its
objective teaches users to depend on that, so the book suggests
occasional planned downtime, throttling, or a system that is no
faster under light load.

The window is longer than the data. A 28-day objective computed on a
Prometheus server that keeps 15 days of samples, the default when no
retention is set, quietly measures 15 days and reports it as four
weeks. Set the retention past the window before trusting the number:

```text
prometheus --storage.tsdb.retention.time=35d
```

Further reading

  * Service Level Objectives — chapter 4 of Google's Site
Reliability Engineering book: indicators, objectives and agreements,
and how to choose a target.

  * Implementing SLOs — the SRE Workbook's chapter on SLIs as
ratios, error budgets, windows and writing the objective down.

  * Example Error Budget Policy — a complete policy for a fictional
game service: when releases stop, and who settles a disagreement.

  * Embracing Risk — why Google measures availability by request
success rather than uptime, and what each step of reliability costs.

  * Availability Table — how much downtime each level of
availability allows per year, quarter, month, week, day and hour.

  * Storage — how Prometheus keeps samples on disk, and the
retention flags that decide how far back an SLO query can see.

hThis page on the web	URL:https://engineer.company/glossary/service-level-objective/	engineer.company	70
