In my recent article on the Philosophy of VoIP Infrastructure, I explored how technical decisions are fundamentally philosophical choices. Today, I want to dive deeper into one of the most critical frameworks for understanding distributed systems: the CAP theorem, and how it applies specifically to Kamailio SIP proxy architectures.

The CAP theorem, formulated by Eric Brewer in 2000, states that a distributed system can only guarantee two out of three properties simultaneously:

• Consistency (C): All nodes see the same data at the same time
• Availability (A): Every request receives a response, without guarantee that it contains the most recent data
• Partition Tolerance (P): The system continues to operate despite network partitions

In telephony, these aren't abstract concepts—they represent real decisions that affect call quality, user experience, and system reliability.

The Telephony Reality: Partition Tolerance is Non-Negotiable

Let's be clear from the start: in telecommunications infrastructure, partition tolerance is mandatory. Network failures happen. Links go down. Datacenters lose connectivity. Your architecture must handle these scenarios.

This means we're really choosing between:

• CP Systems: Consistency + Partition Tolerance (sacrifice Availability)
• AP Systems: Availability + Partition Tolerance (sacrifice Consistency)

There is no CA option in real-world distributed Kamailio deployments.

Kamailio's Architectural Components and CAP Trade-offs

1. User Registration and Location Service

This is where CAP decisions become most visible.

Scenario A: Centralized Location Database (CP approach)

┌─────────────┐
│  Kamailio 1 │─┐
└─────────────┘ │
                 ├──► PostgreSQL Primary
┌─────────────┐ │     (Single Source of Truth)
│  Kamailio 2 │─┘
└─────────────┘

Trade-off: Prioritizes consistency over availability.

-- All Kamailio instances query the same database
SELECT contact, expires FROM location 
WHERE username = 'alice@domain.com';

Consequences:

• ✅ Perfect consistency: All proxies see identical registration state
• ✅ No split-brain scenarios
• ❌ Database becomes single point of failure
• ❌ Network partition to database = no registrations work
• ❌ Higher latency for location lookups

When to choose this:

• Regulatory requirements demand precise registration tracking
• Billing systems need exact call detail records
• Small to medium deployments where database redundancy is manageable

Scenario B: Distributed DMQ with Local Cache (AP approach)

┌─────────────┐         ┌─────────────┐
│  Kamailio 1 │◄───────►│  Kamailio 2 │
│  + DMQ      │   Sync  │  + DMQ      │
│  + htable   │         │  + htable   │
└─────────────┘         └─────────────┘

Trade-off: Prioritizes availability over strong consistency.

# kamailio.cfg
loadmodule "dmq.so"
loadmodule "htable.so"

modparam("htable", "htable", "loc=>size=16;autoexpire=3600;dmqreplicate=1")
modparam("dmq", "server_address", "sip:10.0.1.10:5060")
modparam("dmq", "notification_address", "sip:10.0.1.11:5060")

Consequences:

• ✅ High availability: Each node operates independently
• ✅ Low latency: Local memory lookups
• ✅ Survives network partitions
• ❌ Eventual consistency only
• ❌ Possible temporary registration inconsistencies
• ❌ DMQ synchronization delays

When to choose this:

• High-traffic environments where milliseconds matter
• Geographically distributed installations
• Scenarios where temporary inconsistency is acceptable

2. Dialog State Management

Dialog state (active calls) presents unique CAP challenges.

The Problem

During an active call:

1. User moves between proxies (mobile handoff)
2. Network partition splits your cluster
3. Which proxy "owns" the dialog state?

CP Approach: Shared Dialog Database

modparam("dialog", "db_url", "postgres://kamailio:password@db-cluster/kamailio")
modparam("dialog", "db_mode", 1)  # Real-time database writes
modparam("dialog", "db_update_period", 5)

Trade-off:

• Consistency in dialog state across cluster
• Risk: Database unavailability = no new calls

AP Approach: Distributed Dialog with DMQ

modparam("dialog", "enable_dmq", 1)
modparam("dialog", "dmq_notification_address", "sip:10.0.1.10:5060")

Trade-off:

• Calls survive network partitions
• Risk: Temporary dialog state divergence

Real-world hybrid approach:

# Write to database asynchronously, sync via DMQ immediately
modparam("dialog", "db_mode", 2)  # Write on timer
modparam("dialog", "db_update_period", 60)
modparam("dialog", "enable_dmq", 1)

This gives you:

• Fast DMQ synchronization for active operations
• Database as eventual consistency backup
• Reasonable behavior during partitions

Multi-Datacenter Architectures: CAP at Scale

When spanning multiple datacenters, CAP decisions become existential.

Scenario: Transcontinental Deployment

    US-EAST                      EU-WEST
┌─────────────┐              ┌─────────────┐
│  Kamailio   │              │  Kamailio   │
│  Cluster    │◄────WAN─────►│  Cluster    │
│  + Local DB │              │  + Local DB │
└─────────────┘              └─────────────┘
     ▲                            ▲
     │                            │
  US Users                    EU Users

Option 1: Global Consistency (CP)

Use a globally distributed database like CockroachDB or Galera Cluster with synchronous replication.

modparam("usrloc", "db_url", "mysql://global-cluster/kamailio")
modparam("usrloc", "db_mode", 3)  # Write-through cache

Consequences:

• ✅ Global view of registrations
• ❌ 100-200ms latency on every registration
• ❌ Transatlantic link failure = system down

Option 2: Regional Autonomy (AP)

Each region operates independently with asynchronous synchronization.

# US-EAST kamailio.cfg
modparam("usrloc", "db_url", "mysql://us-east-db/kamailio")
modparam("dmq", "notification_address", "sip:eu-west-dmq:5060")
modparam("dmq", "multi_notify_timeout", 5000)  # Tolerate EU delays

Consequences:

• ✅ Each region fully functional during WAN partition
• ✅ Local latencies (1-5ms)
• ❌ User might appear registered in both regions temporarily
• ❌ Call routing decisions based on stale data

The Philosophical Choice

This is where philosophy meets infrastructure:

Do you value:

• Truth (consistency): Users see accurate state, even if system becomes unavailable
• Responsiveness (availability): System always responds, even if answers are temporarily inaccurate

There's no universally correct answer. It depends on your values and use case.

Practical Decision Framework

Step 1: Identify Your Critical Path

What operations are on the critical path of call setup?

INVITE arrives → Location lookup → Route decision → Forward
              ↑ CRITICAL        ↑ CRITICAL

Make these AP-optimized (local, fast).

Billing record → Database write → CDR generation
              ↑ NOT CRITICAL

Make these CP-optimized (consistent, can be slower).

Step 2: Define Acceptable Inconsistency Windows

Ask: "If data is inconsistent for X seconds, what breaks?"

• User registration: 5-10 seconds usually acceptable
• Active dialog state: 1-2 seconds acceptable
• Billing data: 0 seconds—must be consistent

Step 3: Choose Your Replication Strategy

┌──────────────────┬─────────────┬──────────────┬─────────────┐
│ Data Type        │ Consistency │ Replication  │ Storage     │
├──────────────────┼─────────────┼──────────────┼─────────────┤
│ Registrations    │ Eventual    │ DMQ          │ htable      │
│ Active dialogs   │ Eventual    │ DMQ          │ dialog mod  │
│ Billing CDRs     │ Strong      │ Sync DB      │ PostgreSQL  │
│ Routing rules    │ Strong      │ DB + cache   │ PostgreSQL  │
│ Statistics       │ None        │ Local only   │ Memory      │
└──────────────────┴─────────────┴──────────────┴─────────────┘

Real-World Example: Handling Network Partitions

The Scenario

Your cluster splits into two islands:

Before:                    After partition:
   A ←→ B                     A    B
   ↕    ↕                     ↓    ↓
   C ←→ D                     C    D

CP Behavior

# Only majority partition continues accepting registrations
if (!pike_check_req()) {
    if (!has_quorum()) {
        sl_send_reply("503", "Service Unavailable - No Quorum");
        exit;
    }
}

Nodes A+C (if majority) continue. B+D reject new registrations.

Result: Half your users can't register, but data remains consistent.

AP Behavior

# All nodes continue operating independently
route[REGISTER] {
    save("location");  # Save locally
    dmq_send_message();  # Sync when possible
    sl_send_reply("200", "OK");
}

Result: All users can register, but you might have duplicate registrations temporarily.

Post-Partition Reconciliation

When the partition heals, you need conflict resolution:

# Last-write-wins based on Contact timestamp
route[DMQ_SYNC] {
    if ($var(remote_cseq) > $var(local_cseq)) {
        # Remote is newer, update local
        sql_query("UPDATE location SET contact='$var(remote_contact)' 
                   WHERE username='$rU'");
    }
}

Performance Implications

Benchmarking CP vs AP

In our lab testing with SIPp:

# CP Configuration (Centralized DB)
sipp -sf REGISTER.xml -r 500 -l 10000 -m 50000
  - Average response time: 45ms
  - 99th percentile: 120ms
  - Max throughput: 500 CPS

# AP Configuration (DMQ + htable)  
sipp -sf REGISTER.xml -r 2000 -l 50000 -m 200000
  - Average response time: 8ms
  - 99th percentile: 25ms
  - Max throughput: 2000+ CPS

The AP approach is 5-6x faster but with eventual consistency trade-offs.

Hybrid Architectures: Having Your Cake

The most sophisticated deployments use hybrid approaches:

Tiered Consistency Model

┌─────────────────────────────────────────────┐
│ Tier 1: Real-time (AP)                      │
│  - Registration lookup: DMQ + htable        │
│  - Dialog state: DMQ + dialog module        │
│  - Millisecond latency                      │
├─────────────────────────────────────────────┤
│ Tier 2: Near-real-time (Eventual C)        │
│  - Location backup: Async DB writes         │
│  - Dialog backup: Periodic DB sync          │
│  - Second-level latency                     │
├─────────────────────────────────────────────┤
│ Tier 3: Batch (Strong C)                   │
│  - CDR generation: Sync DB writes           │
│  - Billing records: Transactional           │
│  - Sub-second latency acceptable            │
└─────────────────────────────────────────────┘

Configuration Example

# Route registrations
route[REGISTER] {
    # AP: Save to local htable immediately
    sht_insert("loc=>$au", "$ct");
    
    # AP: Propagate via DMQ for cluster sync
    dmq_send_message();
    
    # Eventual C: Queue for DB write
    mqueue_add("db_writes", "$au|$ct|$Ts");
    
    sl_send_reply("200", "OK");
}

# Background DB sync worker
route[DB_WORKER] {
    while(mqueue_fetch("db_writes")) {
        sql_query("INSERT INTO location ...");
    }
}

When Consistency Actually Matters: Billing

One area where CP is non-negotiable: billing.

route[BYE] {
    # Calculate call duration
    $var(duration) = $TS - $dlg_var(start_time);
    
    # MUST write to database synchronously
    # Financial data cannot be "eventually consistent"
    sql_query("INSERT INTO cdrs 
               (caller, callee, duration, cost) 
               VALUES ('$fU', '$tU', '$var(duration)', '$var(cost)')");
    
    if ($dbr(result) != 1) {
        xlog("L_ERR", "CRITICAL: CDR write failed for call $ci\n");
        # Store in local queue for retry
        mqueue_add("cdr_retry", "$var(cdr_data)");
    }
}

Why CP here?

• Regulatory requirements
• Revenue protection
• Audit trails
• Fraud prevention

Losing a CDR means losing money and potentially violating regulations.

Observability: Monitoring CAP Trade-offs

You need metrics to understand your system's behavior:

# Consistency metrics
- Registration sync lag (DMQ)
- Database replication lag
- Conflicting registrations detected

# Availability metrics  
- Successful registration rate
- Failed DB connections
- Fallback activations

# Partition detection
- DMQ peer connectivity
- Database connection health
- Split-brain events

Sample prometheus metrics:

# Consistency
kamailio_dmq_sync_lag_seconds
kamailio_location_conflicts_total

# Availability
kamailio_registrations_success_total
kamailio_registrations_failed_total
kamailio_db_connection_failures_total

Conclusion: There Are No Perfect Solutions

The CAP theorem isn't a problem to solve—it's a constraint to navigate. In Kamailio architectures:

1. Partition tolerance is mandatory in distributed deployments
2. Choose CP when correctness is critical (billing, compliance)
3. Choose AP when responsiveness is critical (real-time signaling)
4. Use hybrid approaches for different data types
5. Monitor actively to understand actual consistency windows

The "right" architecture depends on your specific requirements, risk tolerance, and philosophical stance on the trade-off between truth and availability.

Remember: Every time you choose a replication strategy, configure a timeout, or design a failover mechanism, you're making a philosophical choice about what your system values most.

Further Reading

• Original CAP theorem paper: Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services
• Kamailio DMQ module documentation
• My article: Philosophy of VoIP Infrastructure
• Designing Data-Intensive Applications by Martin Kleppmann

What CAP trade-offs have you encountered in your telephony infrastructure? Share your experiences in the comments.