The power of flow-based SLOs

written by Sergei Kukhariev

As engineering systems grow, they behave less like simple machines and more like complex ecosystems. The neat boundaries between microservices blur. The most challenging problems often arise not from a single failing component, yet from the unpredictable, emergent behavior of services interacting with each other. In this reality, some monitoring methods can fail us.

Many teams start with threshold-based alerts, getting paged if CPU utilization spikes or memory usage crosses a line. This approach has fundamental flaws. It’s often noisy, waking an on-call engineer for an issue that has no user impact. If the CPU is high while customers are happy, it isn’t a true emergency. Conversely, this method can be blind, missing subtle, cascading failures where every individual service looks green, while the user’s request fails. Thresholds are guesses about what might correlate with a bad user experience. We can do better.

A significant step forward is the adoption of Service Level Objectives (SLOs). SLOs are an improvement because they focus on the user experience. Instead of monitoring machine-centric metrics, we define a Service Level Indicator (SLI) that categorizes events into two simple buckets: “good” and “bad.” The ratio of good events to total events gives us our SLI value. The SLO is our target for that SLI over a period, like 30 days. The allowable amount of failure — the gap between 100% and our SLO target — is our error budget.

We found, however, that even per-service SLOs were not sufficient. An SLO for a single service can be perfectly healthy, while a critical user journey that depends on five such services fails. Observability is most effective when measured as close to the user as is reasonable. This realisation led us to our next evolution: flow-based SLOs.

From service-level to flow-based thinking

In 2023, we began focusing on “Critical Flows” — the essential user journeys in our system. Instead of measuring the performance of an individual service, we decided to measure the success of the entire flow from the user’s perspective.

This approach surpassed all our expectations. It caught issues before any other alert fired, and it was never noisy. The reason is that it measures what actually matters: whether the user can accomplish their goal.

Let’s make this concrete with an example: an outgoing Pix transaction in Brazil, a critical flow we call “Pix Out."

Here is the user journey:

Each step in this journey is an event we consider for our SLO. These steps are not equal in their requirements. A lookup of a recipient’s key in the central bank’s database can tolerate more latency than the transfer initiation itself, which is entirely within our control. A flow-based SLO embraces this nuance.

Here is what our SLO for this flow looks like, expressed in plain language:

99.85% of requests related to Pix Out Critical Flow, measured from BFF point of view, will succeed within the following durations:

Key search — 1500 ms

Transfer Initiation — 900 ms

Check transfer status — 400 ms

This single, holistic objective tells us how we are really doing. It captures the combined health of multiple downstream services as they work together to deliver value.

To give you a more visual example, here’s an example of such an SLO error budget burn in a particularly rough month:

More than a metric: A language for prioritisation

A powerful aspect of SLOs is that they create a shared language across functions. An error budget isn’t just a monitoring metric; it’s a data-driven tool for making business decisions.

  • Burning budget too fast?

    This is a clear signal that we are shipping with poor quality or that reliability is degrading. It’s time to slow down feature development and focus on stability.

  • Never burning the budget?

    We may be acting too cautiously. We have room to take more risks, innovate, and ship features faster.

This framework helps resolve the classic tug-of-war between “new features” and “tech debt.” The error budget, a goal that product and engineering agree on together, tells us where to focus our efforts.

Formalise your response: The Error Budget Policy

To make this framework truly effective, you must decide in advance how you will respond to budget burn. Don’t wait for a crisis to have this discussion. We found that writing an explicit Error Budget Policy was a critical step. This document formalises the agreements on how the team will react to different levels of budget consumption.

Our policy defines clear thresholds for action. For example:

  • When 33% of the budget is burned, two team members stop their current work to investigate.

  • When 66% is burned, three or more engineers swarm on the problem.

  • When the budget is entirely burned, we treat it as a major incident, requiring a full post-mortem and follow-up actions.

This policy removes ambiguity. It ensures that the team responds consistently and appropriately, turning the abstract concept of an error budget into concrete, predictable actions.

A real-world example: Catching the “unknown unknowns”

The true power of this approach became clear during a recent subtle incident. A “slow burn” alert fired for our Pix Out flow, indicating a small yet steady increase in failures. Our service-level dashboards were all green. There were no CPU spikes, no memory alarms, nothing to indicate a problem.

Because the flow SLO alerted us, we knew something was wrong from the user’s perspective. Digging deeper, we discovered that one of our core downstream services had begun to crash and restart its instances one by one. Each individual restart was too fast to trigger a traditional alert, and the service as a whole had enough instances to appear healthy. The flow-based SLO, however, caught the cumulative impact of these tiny, cascading failures on the end-to-end user journey.

This alert fired a full five minutes before any other monitoring caught the issue. Those five minutes were critical. They allowed us to get ahead of the problem, prevent a full-blown outage, and dramatically reduce the impact on our users. This is the unique power of a flow-based SLO: it is good at detecting some of the “unknown unknowns.”

Practical advice for getting started

If you’re ready to explore flow-based SLOs, here is some practical advice from our journey.

How to pick an SLO target

Don’t guess. Start by analysing your historical performance. If your data shows that your flow has achieved 97% reliability over the last three months, setting a target of 99.9% is unrealistic. You have work to do before you can achieve that. Be honest and data-informed.

Also, consider your hard dependencies. The speed of the caravan is the speed of its slowest camel. You cannot achieve four nines of reliability (99.99%) if you depend on a third-party service that only provides two nines (99.0%). Your SLO target must be realistic about the ecosystem in which your flow operates.

When to revisit your SLOs

SLOs are not static. They are living documents that should evolve. We recommend revisiting them periodically, and always under these conditions:

  • Significant usage changes:

    A major increase in traffic might strain the system in new ways.

  • Changes in dependencies:

    A new third-party provider or a significant change in an internal service can alter your reliability profile.

  • New data from incidents:

    Every incident is a learning opportunity. Use that knowledge to refine your SLO.

  • Changes in user expectations:

    As your product matures, users may develop higher expectations for its reliability.

When not to use flow-based SLOs

This approach is powerful, yet it is not a silver bullet. It may not be the right fit in a few scenarios.

  • Unbalanced flows:

    If you are measuring an e-commerce funnel where the conversion rate between stages is very low, a flow-based SLO might not be sensitive enough. The high volume of events at the top of the funnel will drown out the low volume of events at the bottom.

  • Low traffic:

    SLOs in general require a certain scale to be statistically significant. If you have a very low-traffic flow, the natural variance can lead to noisy and unreliable alerts.

  • Teams new to SLOs:

    Flow-based SLOs are a more mature, advanced practice. If your team is beginning its SLO journey, start with simpler service-level SLOs. Use this time to learn your tooling, build confidence in your metrics, and establish the cultural processes around error budgets. Once you are comfortable, you can graduate to the more powerful flow-based approach.

It is also important to state that SLO monitoring does not fully replace traditional monitoring. We still measure the “four golden signals” (Latency, Traffic, Errors, and Saturation) for our individual services. Flow-based SLOs are a powerful layer on top, not a replacement for, foundational monitoring. You can find excellent information about these signals in the Google SRE Book.

Final thoughts

Our experience with flow-based SLOs has been overwhelmingly positive, and my confidence in this approach continues to grow. They provide a nuanced, user-centric view of system health that traditional methods lack. They have helped us catch problems earlier, reduce alert fatigue, and create a shared language for making data-driven decisions about reliability.

For teams just starting their SLO journey, a simpler per-service SLO is the right first step. Flow-based SLOs are an advanced technique. They are, however, a powerful tool that every engineering organisation should have in its toolbox.

If you’d like to chat more about this topic, feel free to reach out.

Resources

Be empowered to do more that matters

Help us build a world where small businesses can thrive

Explore open roles

Inside SumUp's AI hackathon: From idea to prototype in 48 hours

See how 85 SumUppers turned everyday challenges into working AI prototypes in just 48 hours at our Berlin AI Hackathon. No coding required.

Read more

Evaluating an LLM application for generating free-text narratives in financial crime

Explore the effectiveness of an LLM application in generating financial crime narratives and how this technology enhances compliance and risk assessment.

Read more

Polymorphic Double-Label Serialisation

Presents a custom Moshi adapter that supports polymorphic JSON deserialization using one or two discriminator fields, enabling more flexible subtype mapping and graceful handling of unknown variants.

read more