← All Case Studies
Architecture Case Study 04 · Messaging Flow
📨 Message Delivery & Retry Flow
How message-service delivers campaigns via Rocketman, what happens when delivery fails, and why the absence of a dead-letter queue is a confirmed platform risk that amplifies downstream notification traffic.
message-serviceRocketmanKafkaCassandraBigtableincentive-service
📖
Overview
message-service is the CRM messaging hub that manages campaign lifecycle, audience import, and delivery via Rocketman (email) and Regla (push). The critical architectural risk: no dead-letter queue means Rocketman delivery failures trigger a retry amplification cascade — under degradation, message-service becomes a traffic multiplier for an already-failing downstream.
⚙️
Campaign Delivery Lifecycle — 7 Stages
From campaign creation through audience import to Rocketman delivery and retry handling.
1
Campaign Create
Tetris CMS · POST /api/campaign · MySQL campaign metadata
<20ms
→
2
Audience Import
ImportAudienceActor · GCS files · 1min poll · Cassandra/Bigtable write
batch
→
3
Email Trigger
Rocketman polls POST /api/getemailmessages · 500/min limit
12–50ms
→
4
Cache Lookup
EmailCampaignCache (55s) · ActiveCampaignCache · L1 hit
<2ms
→
5
Qualification
Cassandra user assignment lookup · SAD support
10–30ms
→
6
incentive-svc
Discount lookup · sync call · NO circuit breaker ← risk
10–40ms 🔴
→
7
Rocketman Send
Email delivery · 3× retry on 5xx · then silent DROP
variable
💀
No DLQ: failed messages retry 3× then silently drop — zero recovery mechanism
Rocketman delivery failures are retried 3 times then permanently dropped. No dead-letter queue exists. Under Rocketman degradation, retry load amplifies to 3× normal outbound traffic — simultaneously degrading Rocketman further. Confirmed retry amplification cascade.
🔁
Retry Amplification Analysis
🔴
Retry math: 1 Rocketman outage → 3× outbound message-service traffic
1 failed delivery → 3 retry attempts → each retry re-enters same degraded Rocketman endpoint. 1000 concurrent failures = 3000 Rocketman connections while Rocketman is already degraded. Classic thundering herd anti-pattern.
⚠️
ImportAudienceActor runs on ALL batch pods with no distributed lock
6 background jobs run on every batch fleet pod simultaneously. Under N-pod deployment, N×concurrent GCS reads and Cassandra writes hit simultaneously. Confirmed Redis-lock anti-pattern — all 6 jobs should acquire a SETNX distributed lock before executing.
✅
RTA circuit breaker IS configured — correct resilience pattern
Real-time audience (RTCIA) calls in getMessages flow DO have a Resilience4j circuit breaker with graceful fallback. This is the correct pattern — the missing piece is applying the same pattern to the incentive-service call in getEmailMessages.
🔴
Retry amplification is a confirmed blast radius multiplier for Rocketman
When Rocketman degrades, message-service makes the situation worse — not better. Each failed email triggers 3 retries against an already-degraded downstream. Fix: add DLQ topic + max-retries=1 with exponential backoff.
⚠️
incentive-service sync call in getEmailMessages has no circuit breaker
Unlike getMessages (which has RTA circuit breaker), getEmailMessages calls incentive-service synchronously with no fallback. Rocketman polls this endpoint 500/min — a slow incentive-service response extends every email delivery by 10–40ms.
🎯
Key Takeaway
message-service's retry behavior is the platform's most actionable risk: two low-effort fixes (DLQ + circuit breaker) eliminate the retry amplification cascade and the incentive-service P99 extension. Both can be shipped in a single sprint. The audience import no-lock issue (N×concurrent pod writes) should be addressed before the next large campaign launch.