Network Partition

RTO Timeline

Failure Model

Key difference between network partition and node crash:

In network partition scenarios, the primary PostgreSQL may still be running and accepting writes, causing split-brain issues. Patroni solves this through active demotion: when unable to refresh Leader Key, proactively demotes PostgreSQL to read-only or shuts it down.

Timeline Analysis

Phase 1: Primary Demotion

When primary Patroni is network-partitioned from DCS, it cannot refresh Leader Key and starts retrying.

Timeline:
  Partition      Detect partition      Retry timeout      Primary demotes
     |               |                    |                    |
     |←── loop ──→|←── retry ──→|

• Detection delay: After partition occurs, must wait for next loop_wait cycle to detect
• Retry phase: Patroni continuously retries DCS operations during retry_timeout
• Active demotion: After retry timeout, Patroni proactively demotes PostgreSQL (prevents split-brain)

Key design: Patroni requires constraint loop_wait + 2 × retry_timeout ≤ ttl to ensure primary demotes before TTL expires.

Phase 2: Lease Expiration

After primary demotion, Leader Key still exists in DCS, must wait for TTL to naturally expire.

Timeline:
  Primary demoted                   TTL expires
     |                                 |
     |←── ttl - (loop + retry) ──→|

Since the primary has demoted, waiting time during this phase is the remaining TTL time. Since partition detection and remaining TTL are negatively correlated (earlier partition means slower detection but longer remaining TTL), their sum is constant:

Note: Primary demotion + lease expiration total time still approximately equals ttl, same as expire failure.

Phase 3: Replica Detection

Replica wakes up in loop_wait cycle and checks Leader Key status in DCS.

Timeline:
    Lease expired      Replica wakes
       |                  |
       |←── 0~loop ─→|

• Best case: Replica wakes right when lease expires, wait 0
• Worst case: Replica just entered sleep when lease expires, wait loop
• Average case: loop/2

Phase 4: Lock & Promote

After replica discovers Leader Key expired, it starts the election process.

Election flow:
  ReplicaA ──→ Query replication position ──→ Compare ──→ Try lock ──→ Success
  ReplicaB ──→ Query replication position ──→ Compare ──→ Try lock ──→ Fail

• Best case: Single replica or directly acquires lock and promotes, ≈ 0
• Worst case: DCS API call timeout, 2s
• Average case: 1s

Phase 5: Health Check

HAProxy detects new primary coming online, requires rise consecutive successful health checks.

Detection timeline:
  New primary    First check    Second check   Third check (UP)
     |              |               |               |
     |←─ 0~inter ─→|←─ fast ─→|←─ fast ─→|

• Best case: (rise-1) × fastinter
• Worst case: (rise-1) × fastinter + inter
• Average case: (rise-1) × fastinter + inter/2

RTO Formula

Sum all phase times to get total RTO.

Since primary demotion + lease expiration ≈ ttl, network partition RTO formula is same as expire failure:

Best Case

Average Case

Worst Case

Model Calculation

Substituting the four RTO model parameters into the formulas:

pg_rto_plan:  # [ttl, loop, retry, start, margin, inter, fastinter, downinter, rise, fall]
  fast: [ 20  ,5  ,5  ,15 ,5  ,'1s' ,'0.5s' ,'1s' ,3 ,3 ]  # rto < 30s
  norm: [ 30  ,5  ,10 ,25 ,5  ,'2s' ,'1s'   ,'2s' ,3 ,3 ]  # rto < 45s
  safe: [ 60  ,10 ,20 ,45 ,10 ,'3s' ,'1.5s' ,'3s' ,3 ,3 ]  # rto < 90s
  wide: [ 120 ,20 ,30 ,95 ,15 ,'4s' ,'2s'   ,'4s' ,3 ,3 ]  # rto < 150s

Patroni constraint validation (loop + 2×retry ≤ ttl):

Four mode calculation results (seconds, format: min / avg / max)

Conclusion: Network partition RTO is same as expire failure (node crash), as the bottleneck is TTL expiration time.

Split-brain Protection

The biggest risk of network partition is split-brain: old primary may still be running and accepting writes. Patroni provides multiple protection mechanisms:

1. Primary Self-Demotion

Patroni’s core protection mechanism: when unable to refresh Leader Key, proactively demotes PostgreSQL.

# Patroni pseudo-code logic
if not can_refresh_leader_key():
    retry_until(retry_timeout)
    if still_cannot_refresh():
        demote_postgresql()  # Demote to read-only or shut down

2. Linux Watchdog

If Patroni process hangs and cannot execute demotion, Linux watchdog will force system restart.

# patroni.yml configuration
watchdog:
  mode: required  # Require watchdog available
  device: /dev/watchdog
  safety_margin: 5

3. Fencing Mechanism

Can configure fencing scripts to forcibly isolate old primary (e.g., disable network interface, stop service, etc.).

Special Scenarios

Scenario A: Primary partitioned from DCS, replicas normal

This is the most common network partition scenario, the main focus of this article.

┌─────────┐         ╳         ┌─────────┐
│ Primary │ ←── Partition ──→ │   DCS   │
│ Patroni │                   │  etcd   │
└─────────┘                   └─────────┘
                                  ↑
                              Normal connection
                                  ↓
                              ┌─────────┐
                              │ Replica │
                              │ Patroni │
                              └─────────┘

• Primary Patroni cannot refresh Leader Key → Active demotion
• Replica normally detects TTL expiration → Elected as new primary
• RTO ≈ Expire failure RTO

Scenario B: Primary normal, replica partitioned from DCS

┌─────────┐                   ┌─────────┐
│ Primary │ ←── Normal ──→    │   DCS   │
│ Patroni │                   │  etcd   │
└─────────┘                   └─────────┘
                                  ╳
                              Partition
                                  ╳
                              ┌─────────┐
                              │ Replica │
                              │ Patroni │
                              └─────────┘

• Primary normally refreshes Leader Key
• Replica cannot participate in election (but replication can continue)
• No failover triggered, service continues normally

Scenario C: All nodes partitioned from DCS

┌─────────┐         ╳         ┌─────────┐
│ Primary │ ←── Partition ──→ │   DCS   │
│ Patroni │                   │  etcd   │
└─────────┘                   └─────────┘
                                  ╳
┌─────────┐         ╳             │
│ Replica │ ←── Partition ────────┘
│ Patroni │
└─────────┘

• Primary demotes, replica cannot elect
• Cluster completely unavailable
• Requires manual intervention to restore DCS connectivity

Comparison with Other Failures

Key Insight: Network partition RTO is same as expire failure, but requires additional split-brain protection mechanisms. Ensuring loop_wait + 2 × retry_timeout ≤ ttl constraint is the key design to prevent split-brain.