Rewriting the 'RPO and RTO' page to clear up common confusion #4091
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
📖 Docs PR preview links
|
lukeknep
changed the title
lukeknep
commented
Dec 23, 2025
| Temporal Cloud is designed to limit data loss after recovery when the incident triggering the failover is resolved. | ||
| - "Temporal-initiated failovers:" Also known as "automatic failovers," these failovers are initiated by Temporal's tooling and/or on-call engineers on Namespaces that have High Availability enabled. **Temporal highly recommends keeping Temporal-initiated failovers enabled,** which is the default for all Namespaces with High Availability features. Users can still trigger manual failovers on their Namespaces even if Temporal-initiated failovers are enabled. When Temporal-initiated failovers are disabled on a Namespace, Temporal's RTO for that Namespace is unbounded (it is dependent on how long the underlying outage lasts) | ||
|
|
||
| Temporal Cloud strives to maintain a P95 [replication lag](/cloud/high-availability/monitoring#replication-lag-metric) of less than 1 minute. |
Contributor
Author
There was a problem hiding this comment.
I removed this bit because
- I'm not sure p95 is good enough. This could be read as "up to 5% of Namespaces could be above the 1-minute RPO at any given moment."
- We already say we have a 1-minute RPO. I don't think we need additional standards / goals to be publicly stated. They would only add confusion. Let's state our main goal (RPO) and stand by it.
bechols
approved these changes
Dec 23, 2025
|
|
||
| Recovery Point Objective (RPO) and Recovery Time Objective (RTO) define data durability and service restoration timelines, respectively. | ||
| In Temporal Cloud, these objectives vary depending on your deployment configuration and the scope of any failure. | ||
| Recovery Point Objective (RPO) and Recovery Time Objective (RTO) are the objectives that Temporal strives to meet for data durability (recovery point) and service restoration (recovery time) in the event of cloud outages. |
Contributor
There was a problem hiding this comment.
RPO and RTO are how we measure, and low values for RPO and RTO are what we strive for. Could tighten this phrasing.
Contributor
Author
There was a problem hiding this comment.
That's not accurate (or at least, if I understand your comment correctly, that's not how the terms are used in the industry)
- Recovery Point/Time "Objective": this is the goal we have for all outages. That's why the term has "Objective" in it's name.
- recovery time / recovery point: this is the actual observed values in a given outage. I could say "observed recovery time" or "achieved recovery time," but that gets bloated.
I wanted to make the distinction between the two terms really clear in the doc. If it's not clear, then I need to reword it.
P.S. Confirmed the industry standard with GPT 5.1:
Contributor
There was a problem hiding this comment.
To me, the current wording boils down to "we strive for RPO", which doesn't really have any informational content without the actual numerical objective. "We strive for zero RPO" or "we strive for sub 20 minute RTO" is informative.
Trying this wording with a similar concept: "Uptime is the objective that Temporal strives to meet for availability (service accessibility)". I think it's clearer/more informative to say something like "Temporal Cloud measures availability in terms of service uptime, and has a 99.99% availability SLO and 99.% availability SLA"
All that said: happy to merge as-is.
| Recovery Point Objective (RPO) and Recovery Time Objective (RTO) define data durability and service restoration timelines, respectively. | ||
| In Temporal Cloud, these objectives vary depending on your deployment configuration and the scope of any failure. | ||
| Recovery Point Objective (RPO) and Recovery Time Objective (RTO) are the objectives that Temporal strives to meet for data durability (recovery point) and service restoration (recovery time) in the event of cloud outages. | ||
| These objectives are high priority goals for Temporal Cloud, but are not a contractual commitment. |
Contributor
There was a problem hiding this comment.
This sounds rough. Can we say RPO + RTO aren't part of the availability SLA instead (and link to the SLA page)?
Contributor
There was a problem hiding this comment.
Suggested change
| These objectives are high priority goals for Temporal Cloud, but are not a contractual commitment. | |
| Temporal Cloud's RPO and RTO are complementary to but separate from the [availability SLA](/cloud/sla)." |
| In case of an outage in the active region or cell, Temporal Cloud will failover to the replica so that existing Workflow Executions will continue to run and new Executions can be started. | ||
|
|
||
| ## High Availability, Regional Failure | ||
| The Recovery Point Objective and Recovery Time Objective for Temporal Cloud depend on the type of outage and which [High Availability](/cloud/high-availability) feature your Namespace has enabled: |
| Temporal has put extensive work into tools and processes that minimize the recovery point and achieve its RPO for Temporal-initiated failovers, including: | ||
|
|
||
| **Recovery Time Objective (RTO) - 20 minutes** | ||
| - Best-in-class data replication technology that keeps the replica up to date with the active. |
Contributor
There was a problem hiding this comment.
Could link to https://www.youtube.com/watch?v=mULBvv83dYM where Liang gets into more specifics
|
|
||
| **All writes to storage are synchronously replicated across AZs**, including our writes to ElasticSearch. | ||
| ElasticSearch is eventually consistent, but this does not impact our RPO as there is no data loss. | ||
| - You can detect outages that Temporal doesn't. In the cloud, regional outages never affect every service the same way. It's possible that Temporal--and the services it depends on--are unaffected by the outage, even while your Workers or other cloud infrastructure are disrupted. If you monitor each service in your critical path and alert on unusual |
Contributor
There was a problem hiding this comment.
Can we suggest or link to specific guidance on how to do this?
Co-authored-by: Ben Echols <[email protected]>
Co-authored-by: Ben Echols <[email protected]>
lukeknep
changed the title
sergeybykov
approved these changes
Jan 20, 2026
Member
sergeybykov
left a comment
sergeybykov
left a comment
There was a problem hiding this comment.
A general good practice is to start each sentence on a new line, so that a later reviewer can add a comment to a specific sentence.
| Temporal Cloud delivers different RPO/RTOs based on these scenarios because of the way our platform performs writes to our data provider. | ||
| As Workflows progress in the active region, history events are asynchronously replicated to the replica. | ||
| Because replication is asynchronous, High Availability does not impact the latency or throughput of Workflow Executions in the active region. | ||
| If an outage hits the active region or cell, Temporal Cloud will failover to the replica so that existing Workflow Executions will continue to run and new Workflow Executions can be started. |
Member
There was a problem hiding this comment.
"failover" -> "fail over" (verb vs. noun)
|
|
||
| This means there is _no_ logical corruption and restoration is done from a live replicated instance. | ||
| This applies for both single region Namespaces and multi region Namespaces. | ||
| - Regular drills where we failover our internal Namespaces to test our tooling. |
Member
There was a problem hiding this comment.
"failover" -> "fail over" (verb vs. noun)
| - You can detect outages that Temporal doesn't. In the cloud, regional outages never affect every service the same way. It's possible that Temporal--and the services it depends on--are unaffected by the outage, even while your Workers or other cloud infrastructure are disrupted. If you monitor each service in your critical path and alert on unusual | ||
|
|
||
| **Recovery Time Objective (RTO) - 0** | ||
| - You can sequence your failovers in a particular order. Your cloud infrastructure probably contains more pieces than just your Temporal Namespace: Temporal Workers, compute pools, data stores, and other cloud services. If you manually failover, you can choose the order in which these pieces switch to the replica region. You can then test that ordering with failover drills and ensure it executes smoothly without data consistency issues or bottlenecks. |
Member
There was a problem hiding this comment.
"If you manually failover" -> "If you manually fail over" (verb vs. noun)
|
|
||
| Temporal is active-active across AZs. | ||
| The RTO is stated to be zero, meaning there should be no downtime in such scenarios. | ||
| - You can proactively failover more aggressively than Temporal. While the 20-minute RTO should be sufficient for most use cases, some may strive to hit an even lower RTO. For workloads like high frequency trading, auctions, or popular sporting events, an outage at the wrong time could cause tremendous lost revenue per minute. You can adopt a posture that fails over more eagerly than Temporal does. For example, you could trigger a manual failover at the first sign of a possible disruption, before knowing whether there's a true regional outage. |
Member
There was a problem hiding this comment.
"proactively failover" -> "proactively fail over" (verb vs. noun)
| The RTO is stated to be zero, meaning there should be no downtime in such scenarios. | ||
| - You can proactively failover more aggressively than Temporal. While the 20-minute RTO should be sufficient for most use cases, some may strive to hit an even lower RTO. For workloads like high frequency trading, auctions, or popular sporting events, an outage at the wrong time could cause tremendous lost revenue per minute. You can adopt a posture that fails over more eagerly than Temporal does. For example, you could trigger a manual failover at the first sign of a possible disruption, before knowing whether there's a true regional outage. | ||
|
|
||
| - Even if you have robust tooling to detect an outage and trigger a failover, leaving Temporal-initiated failovers enabled provides a "safety net" in case your automation misses an outage. It also gives Temporal leeway to preemptively failover your Namespace if we detect that it may be disrupted soon, e.g., by a rolling failure that has impacted other Namespaces but not yours, yet. |
Member
There was a problem hiding this comment.
"preemptively failover" -> "preemptively fail over" (verb vs. noun)
|
|
||
| - Namespace 1_A was in the region and its cell experienced a partial degradation that caused 10% of requests to fail in the first 5 minutes, 25% in the second five minutes, and 50% in the third five minutes. Since it was significantly impacted from 10:00:00 to 10:15:00, its Recovery Time was 15 minutes. If it had no other service errors that month, then its service error rate for the month would be: ( (1 - 10%) + (1 - 25%) + (1 - 50%) + 8925 * 100% ) / 8928 = 99.990%. (Note: there are 8928 5-minute periods in a 31-day month.) | ||
|
|
||
| - Namespace 1_B was in the same cell as Namespace 2_A, so it also experienced a partial degradation that caused 10% of requests to fail. However, its owner detected the outage via their own tooling and decided to manually failover at 10:05:00. This Namespace achieved a recovery time of 5 minutes and a service error rate of ( 1 * (1 - 10%) + 8927 * 100% ) / 8928 = 99.998%. |
Member
There was a problem hiding this comment.
"failover" -> "fail over" (verb vs. noun)
|
|
||
| - Namespace 2_A was in the region and its cell was fully network partitioned at the start of the outage, causing 100% of requests to fail. Since it was significantly impacted from 10:00:00 to 10:15:00, its Recovery Time was 15 minutes. If it had no other service errors that month, then its service error rate for the month would be: ( 3 * (1 - 100%) + 8928 * 100% ) / 8640 5-minute periods per month = 99.97%. | ||
|
|
||
| - Namespace 2_B was in the region and was fully network partitioned, causing 100% of requests to fail. However, its owner detected the outage via their own tooling and decided to manually failover at 10:05:00. This Namespace achieved a recovery time of 5 minutes and a service error rate of ( 1 5-minute periods * (1 - 100%) + 8639 5-minute periods * 100% ) / 8640 5-minute periods per month = 99.99%. |
Member
There was a problem hiding this comment.
"failover" -> "fail over" (verb vs. noun)
|
|
||
| - Namespace 2_B was in the region and was fully network partitioned, causing 100% of requests to fail. However, its owner detected the outage via their own tooling and decided to manually failover at 10:05:00. This Namespace achieved a recovery time of 5 minutes and a service error rate of ( 1 5-minute periods * (1 - 100%) + 8639 5-minute periods * 100% ) / 8640 5-minute periods per month = 99.99%. | ||
|
|
||
| All of the above Namespaces were in the affected region, but they achieved varying recovery times and service error rates. |
Member
There was a problem hiding this comment.
Can we say that in all these cases RPO was zero?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Corrects errors and clears up common confusion points around our RPO, RTO, and SLA.
Internal Note on the previously-stated 8-hour RTO / RPO for non-HA Namespaces: