Messaging Service

A distributed, provider-centric messaging platform built on Akka.NET with cluster sharding, Redis-backed message queues, and dynamic per-provider throttling.

Overview

The Messaging Service is a high-throughput, multi-channel messaging platform designed around providers as the central abstraction. Messages are addressed to a specific provider + sender combination, enqueued in Redis, and dispatched through provider-specific worker pools with token-bucket rate limiting and end-to-end backpressure.

Note: This is the second-generation architecture. The previous campaign/message-queue model has been replaced with a lighter-weight provider queue model where message dispatch is driven from Redis rather than database-pulled message batches.

Message Lifecycle & Event Model

Every message progresses through a well-defined lifecycle, producing an append-only event trail in the message_events table. Each event records the canonical status, a substatus (delivery code for sent, error code for failed), and the actual time the event occurred (distinct from when it was persisted to the database).

   ┌──────────────────┐
   │  API enqueues    │── message-enqueued (Status: Ready)
   └────────┬─────────┘
            │
   ┌────────▼─────────┐
   │  Dispatched      │── message-dispatched (Status: Dispatched)
   │  to provider     │
   └────────┬─────────┘
            │
     ┌──────┴──────┐
     │             │
┌────▼────┐   ┌────▼──────┐
│ Provider│   │  HTTP /   │
│ confirms│   │  Timeout  │
│  sent   │   │  Failure  │
└────┬────┘   └────┬──────┘
     │             │
┌────▼────┐   ┌────▼──────┐
│message- │   │ message-  │
│  sent   │   │  failed   │
│ Sent,   │   │ Failed,   │
│ delivery│   │ error_code│
│  _code  │   │           │
└────┬────┘   └───────────┘
     │
┌────▼────────┐
│ message-    │
│  delivered  │
│ Sent,"deliv-│
│ ered"       │
└─────────────┘
Event Type Canonical Status Substatus Column Description
message-enqueued Ready — Message accepted by the API, added to Redis dispatch queue.
message-dispatched Dispatched — Successfully sent to the provider's API, awaiting delivery confirmation.
message-sent Sent delivery_code Provider confirmed the message was accepted (intermediate status). Delivery code is null when no delivery confirmation is available yet.
message-delivered Sent delivery_code Provider confirmed final delivery outcome. delivery_code = "delivered", "undelivered", etc.
message-failed Failed provider_error_code Message could not be sent. Error code = "http_429", "timeout", "circuit_open", etc.

This model allows queries without JSON parsing: the message_status, delivery_code, and provider_error_code columns are indexed and filterable directly via the query API.

Key Concepts

Concept Description
Actor The fundamental unit of computation in Akka.NET. Actors encapsulate state and behavior, communicate exclusively via asynchronous messages, and process one message at a time. This guarantees thread-safety without the need for locks.
Provider Config Provider-level credentials (API keys, tokens) stored in the provider_configs table. Examples: a Twilio account, a Trumpia account, a Meta WhatsApp Business profile.
Provider Sender A specific sending identity (phone number, WhatsApp sender ID, email address) owned by a provider config. Each sender can have its own rate limit (max_messages_per_second), inbound webhook URL, and delivery webhook URL.
Provider Key Composite key (ProviderId, SenderAddress) that uniquely identifies a dispatch target. Used as the shard identity in the Akka cluster.
Message Queue A per-provider-key FIFO queue stored in Redis. Messages are added via API and consumed by an actor shard.
Message Group A logical grouping of messages defined by a CorrelationId. The system aggregates real-time delivery statistics (dispatched, sent, failed, unknown) per group and can push updates via webhooks.

Architecture

Actor System

The service runs as a multi-node Akka.NET cluster using Cluster Sharding. Sharding is a mechanism that distributes actors across the nodes in a cluster, ensuring that for any given unique ID (like a ProviderKey), there is exactly one actor instance running in the entire cluster at any time. This prevents race conditions when reading from queues and automatically redistributes workload if a node crashes or scales horizontally.

The cluster operates with three roles:

Akka Role Responsibilities
api Hosts the HTTP API endpoints (send messages, control queues, webhooks).
dispatcher Runs the ProviderQueueCoordinatorActor shard — dequeues messages from Redis and routes them to provider-specific workers while enforcing rate limits.
poller Runs the PollerManagerActor and ProviderPollerCoordinatorActor shard — scans for pending delivery receipts and polls provider APIs.

A single node can serve multiple roles. Configuration is controlled by the AkkaSettings:Roles setting.

Actor Directory

The system is composed of several specialized actors that handle distinct parts of the messaging lifecycle:

Actor Type Responsibility
ProviderQueueCoordinatorActor Shard Manages the dispatch lifecycle for a specific ProviderKey, fetching batches from Redis and pushing them through the throttle pipeline.
DynamicThrottleActor Child Implements token-bucket rate limiting to enforce MaxMessagesPerSecond and signals backpressure to the coordinator.
WorkerActor Child Pool Provider-specific HTTP workers (e.g., TwilioWorkerActor) that execute the actual API calls to external providers.
PollerManagerActor Singleton Periodically scans Redis for active provider keys and wakes up their respective polling coordinators.
ProviderPollerCoordinatorActor Shard Manages the polling lifecycle for a specific ProviderKey.
ReceiptPollerActor Child Provider-specific HTTP workers that poll external APIs for missing delivery receipts.
EventBatchWriterActor Root Buffers message events (dispatched, delivered, failed) and bulk-inserts them into PostgreSQL.
WebhookNotificationActor Root Pushes delivery receipt webhooks to sender-configured callback URLs.
WebhookJournalWriterActor Root Buffers outbound webhook attempts and bulk-inserts them into PostgreSQL.
GroupStatsSnapshotActor Root Periodically scans Redis for idle message groups and saves their aggregated stats to PostgreSQL.
GroupWebhookActor Root Pushes real-time group stats to client webhooks with at-most-once rate limiting. 
graph TB
    subgraph "HTTP Layer (api role)"
        API[Messaging API<br/>POST /api/v1/messages/send<br/>POST /api/v1/provider-queues/.../pause]
        WEBHOOKS[Webhook Endpoints<br/>/api/webhooks/twilio<br/>/api/webhooks/trumpia]
    end

    subgraph "Redis"
        DISPATCH_QUEUE[Provider Message Queues<br/>Sorted Sets per ProviderKey]
        DISPATCHED_INDEX[Dispatched Messages<br/>Hash + Sorted Set per ProviderKey]
    end

    subgraph "Akka Cluster"
        subgraph "Dispatch Shard (dispatcher role)"
            COORDINATOR[ProviderQueueCoordinatorActor<br/>Sharded by ProviderKey]
            THROTTLE[DynamicThrottleActor<br/>Token Bucket per provider]
            WORKERS[WorkerActor Pool<br/>Round-robin per provider]
        end

        subgraph "Poll Shard (poller role)"
            POLLER_MGR[PollerManagerActor<br/>Scans Redis]
            POLLER_COORD[ProviderPollerCoordinatorActor<br/>Sharded by ProviderKey]
            POLLER[ReceiptPollerActor<br/>Provider-specific HTTP poll]
        end

        subgraph "Core Actors (all roles)"
            DB_WRITER[EventBatchWriterActor<br/>Batched DB writes]
            WEBHOOK_WRITER[WebhookJournalWriterActor<br/>Batched webhook journal writes]
            SNAPSHOT[GroupStatsSnapshotActor<br/>PostgreSQL snapshots]
            WEBHOOK_ACTOR[GroupWebhookActor<br/>Group stats push]
        end
    end

    subgraph "PostgreSQL"
        MSG_EVENTS[message_events]
        CONFIGS[provider_configs]
        SENDERS[provider_senders]
        TRANSACTIONS[message_transactions]
    end

    subgraph "Admin Website"
        WEBSITE[MessagingService.Website<br/>Blazor + API]
    end

    API -->|Enqueue| DISPATCH_QUEUE
    API -->|WakeUp signal| COORDINATOR
    COORDINATOR -->|Fetch batch| DISPATCH_QUEUE
    COORDINATOR -->|SendMessage| THROTTLE
    THROTTLE -->|Forward| WORKERS
    WORKERS -->|HTTP POST| PROVIDERS[External Provider APIs]
    WORKERS -->|EventStream| DB_WRITER

    WEBHOOKS -->|DeliveryReceipt| DB_WRITER
    WEBHOOKS -->|WebhookNotification| WEBHOOK_WRITER

    POLLER_MGR -->|Scan active providers| DISPATCHED_INDEX
    POLLER_MGR -->|WakeUp| POLLER_COORD
    POLLER_COORD -->|Create| POLLER
    POLLER -->|Poll receipts| PROVIDERS
    POLLER -->|EventStream| DB_WRITER

    DB_WRITER -->|Batch flush| MSG_EVENTS

    WEBSITE -.->|CRUD| CONFIGS
    WEBSITE -.->|CRUD| SENDERS
    WEBSITE -.->|Read| TRANSACTIONS

Actor Lifecycle & Supervision

Akka.NET enforces strict parent-child supervision and lifecycle management to ensure resource efficiency and fault tolerance:

  • Dynamic Cleanup (Passivation): To conserve memory, shard coordinators (ProviderQueueCoordinatorActor and ProviderPollerCoordinatorActor) are automatically passivated (stopped) by the cluster if they remain idle past ProviderMaxIdleTimeInSeconds. Because of Akka's parent-child hierarchy, stopping a coordinator recursively stops all of its active children (e.g., the dynamic throttle actor, worker pools, and provider-specific poller actors).
  • Persistent Core Actors: In contrast to the dynamically spawned provider actors, core system actors (EventBatchWriterActor, WebhookJournalWriterActor, GroupStatsSnapshotActor, GroupWebhookActor) are spawned at the root level of the actor system when the node starts. They are node-level singletons (one instance per node) that run continuously to handle global tasks like batched database writes, regardless of the node's configured roles.

API Authentication & Multi-Tenancy

The API authenticates incoming HTTP requests (such as sending messages or querying group stats) using a custom ApiKeyAuthenticationHandler. Clients must provide two HTTP headers:

  • x-client-id: The Tenant ID (a GUID).
  • x-client-secret: The client's raw secret string.

To minimize latency and database load during high-throughput dispatching, the authentication handler uses an IMemoryCache to temporarily store successful verification results (defaulting to 5 minutes). This caching layer prevents the system from querying PostgreSQL and performing expensive PBKDF2 hash comparisons on every single incoming API request.

Data Flow — Message Dispatch

sequenceDiagram
    participant Client
    participant API as POST /api/v1/messages/send
    participant Redis as Redis Queue
    participant Coordinator as ProviderQueueCoordinatorActor
    participant OptOut as IOptOutService
    participant Throttle as DynamicThrottleActor
    participant Worker as WorkerActor Pool
    participant DbWriter as EventBatchWriterActor
    participant DB as PostgreSQL

    Client->>API: POST { providerId, senderAddress, body, ... }
    API->>Redis: RPUSH provider-queue:{providerKey} [message]
    API->>EventBatchWriter: Publish(MessageEnqueued) — via Redis Stream + Akka EventStream
    API->>Coordinator: Tell(WakeUp)
    API-->>Client: 202 { id }

    Coordinator->>Coordinator: ScheduleFetch()
    Coordinator->>Redis: GetMessagesToDispatchAsync(providerKey, batchSize)
    Redis-->>Coordinator: [Message, Message, ...]
    
    Coordinator->>OptOut: GetOptedOutNumbersAsync(sender, [recipients])
    OptOut-->>Coordinator: [optedOutNumber, ...]
    
    loop each message
        alt Recipient opted out
            Coordinator->>DbWriter: Tell(MessageFailed) (Status: OPTOUT)
        else Recipient valid
            Coordinator->>Throttle: Tell(SendMessage)
        end
    end

    alt Token available
        Throttle->>Worker: Forward message
        Worker->>Provider API: HTTP POST
        Provider API-->>Worker: 200 OK (transactionId)
        Worker->>DbWriter: Tell(MessageDispatched)
        DbWriter->>DB: Batch flush (every WriterFlushIntervalInSeconds)
    else Queue full / HighWatermark breached
        Throttle->>Coordinator: Tell(SlowDown)
        Coordinator->>Coordinator: Pause fetch timer
        Note over Coordinator,Throttle: Backpressure active
    end

Data Flow — Delivery Receipts (Webhooks & Polling)

The platform supports two mechanisms for receiving delivery receipts: pushing via provider webhooks (preferred) and active polling (fallback).

sequenceDiagram
    participant Provider as External Provider API
    participant Webhook as Webhook Endpoint
    participant Poller as ReceiptPollerActor
    participant Redis as Redis Tracking Keys
    participant DbWriter as EventBatchWriterActor
    participant Notifier as WebhookNotificationActor
    participant Journal as WebhookJournalWriterActor
    participant Client as Client Callback URL

    %% Webhook Path
    alt Delivery via Webhook (Push)
        Provider->>Webhook: POST /api/webhooks/{provider} (Signature Validated)
        Webhook->>DbWriter: Tell(DeliveryReceiptReceived) (Publishes to EventStream)
    end

    %% Polling Path
    alt Delivery via Polling (Pull)
        Poller->>Redis: Query pending-receipts-poll-index
        Redis-->>Poller: [providerMessageId, ...]
        Poller->>Provider: GET /receipts?ids=...
        Provider-->>Poller: [Receipt, ...]
        Poller->>DbWriter: Tell(DeliveryReceiptReceived) (Publishes to EventStream)
    end

    %% Persistence & Notification (Common)
    Note over DbWriter, Client: EventStream subscriptions handle the receipt asynchronously
    
    par Database Persistence (FlushPipeline)
        DbWriter->>DbWriter: Buffer event
        DbWriter->>PostgreSQL: Batch flush via COPY to message_events
        DbWriter->>Redis: RemoveFromPollIndexAsync(transactionId)
    and Client Notification
        DbWriter-->>Notifier: (Listens to EventStream)
        Notifier->>Client: POST delivery payload to callbackUrl
        Client-->>Notifier: 200 OK
        Notifier->>Journal: Tell(WebhookNotificationRecord)
        Journal->>PostgreSQL: Batch flush via COPY to webhook_notifications
    end

The EventBatchWriterActor & FlushPipeline

The EventBatchWriterActor is a critical singleton actor in the architecture. Rather than writing to PostgreSQL on every individual messaging event (which would overwhelm the database under high load), this actor acts as an asynchronous write-ahead buffer in memory.

It subscribes to the Akka EventStream for three types of events:

  1. Enqueues (MessageEnqueued) — sent by the API when a message is accepted and added to the dispatch queue.
  2. Dispatches (MessageDispatched) — sent by worker actors after a successful HTTP call.
  3. Failures (MessageFailed) — sent by worker actors after an unsuccessful HTTP call, or by pollers on timeout.
  4. Delivery Receipts (DeliveryReceiptReceived) — forwarded from webhook endpoints and poller actors.

Every WriterFlushIntervalInSeconds (or when the internal buffer exceeds EventBatchWriterMaxPendingBeforeForceFlush), the actor triggers the scoped IFlushPipeline to perform the following in a single transaction:

  1. Receipt Deduplication: Ensures multiple webhooks for the same transaction don't duplicate events.
  2. Classification: Converts the raw events into structured MessageEvent objects with the canonical MessageStatus, DeliveryCode (substatus for sent), and ProviderErrorCode (substatus for failed) — all derived from the source event data rather than requiring JSON parsing at query time.
  3. Database Bulk Insert: Executes a highly optimized batched flush via PostgreSQL COPY for the raw event history.
  4. Group Statistics Aggregation: Calculates the Dispatched, Sent, Failed, and Unknown deltas for any messages that belong to a CorrelationId group.
  5. Redis Cache Updates:
    • Uses HINCRBY to apply the calculated deltas to the real-time group-stats Hash in Redis.
    • Calls RemoveFromPollIndexAsync to delete processed receipts from the polling queue (while leaving them in the timeout/payload indices for subsequent webhooks).
  6. Notification Publishing: For every group that had statistics changed, it publishes an EventsWriterEvents.GroupStatsUpdated event back to the EventStream. The GroupWebhookActor listens to this event to push real-time webhooks back to the client.

By decoupling the high-throughput incoming events from the actual database writes, the EventBatchWriterActor ensures the system remains highly responsive and prevents database bottlenecks.

Data Flow — Provider Queue Control

Endpoint Action
POST /api/v1/provider-queues/{providerId}/senders/{senderAddress}/pause Pauses the coordinator for this provider key. In-flight messages complete; no new batches are fetched from Redis.
POST /api/v1/provider-queues/{providerId}/senders/{senderAddress}/resume Resumes batch fetching.
POST /api/v1/provider-queues/{providerId}/senders/{senderAddress}/stop Fire-and-forget stop for the coordinator. Messages already in Redis remain; no new dispatches begin.

All control endpoints return 202 Accepted. Pause and Resume wait up to 5 seconds for the actor to confirm the state transition.

API Reference

Send a Single Message

POST /api/v1/messages/send
X-Api-Key: <key>
Content-Type: application/json

{
  "providerId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "senderAddress": "+15550001111",
  "destinationAddress": "+15559998888",
  "body": "Hello from the platform!",
  "mediaUrls": ["https://cdn.example.com/photo.jpg"],
  "correlationId": "my-ref-42",
  "callbackUrl": "https://my-server/hooks/status"
}

// 202 Accepted — single message returns the assigned message ID
{ "id": "550e8400-e29b-41d4-a716-446655440000" }

The message is enqueued in Redis and a WakeUp signal is sent to the coordinator shard for the matching providerKey.

Message Group Tracking

Clients can group messages into campaigns or logical groups using the CorrelationId field in the send API. The messaging engine automatically tracks aggregated real-time statistics (dispatched, sent, failed, unknown) per group in Redis.

POST /api/v1/groups
X-Api-Key: <key>
Content-Type: application/json

{
  "correlationId": "my-campaign-123",
  "callbackUrl": "https://my-server.com/webhooks/group-stats"
}

// 201 Created
{
  "correlationId": "my-campaign-123",
  "tenantId": "...",
  "callbackUrl": "https://my-server.com/webhooks/group-stats",
  "createdAt": "2023-10-01T10:00:00Z",
  "updatedAt": "2023-10-01T10:00:00Z",
  "dispatched": { "count": 0 },
  "sent": { "count": 0, "substatuses": {} },
  "failed": { "count": 0, "substatuses": {} },
  "unknown": { "count": 0 }
}

GET /api/v1/groups/my-campaign-123/stats
X-Api-Key: <key>

Group stats are tracked in a Redis Hash. When stats are modified, a GroupStatsUpdated event triggers the GroupWebhookActor to push the stats via an HTTP POST to the registered callbackUrl, throttled using Redis SETNX (at-most-once per configured interval). Idle groups are eventually snapshotted into PostgreSQL by the GroupStatsSnapshotActor to preserve history.

Query Message Events

GET /api/v1/events?correlationId=my-campaign-123&page=1&pageSize=50
X-Api-Key: <key>

// 200 OK
{
  "items": [
    {
      "eventId": "550e8400-e29b-41d4-a716-446655440000",
      "outboundMessageId": "6ba7b810-9dad-11d1-80b4-00c04fd430c8",
      "eventType": "message-enqueued",
      "status": "Ready",
      "deliveryCode": null,
      "providerErrorCode": null,
      "correlationId": "my-campaign-123",
      "occurredAt": "2026-06-29T12:34:50Z"
    },
    {
      "eventId": "550e8400-e29b-41d4-a716-446655440001",
      "outboundMessageId": "6ba7b810-9dad-11d1-80b4-00c04fd430c8",
      "eventType": "message-dispatched",
      "status": "Dispatched",
      "deliveryCode": null,
      "providerErrorCode": null,
      "correlationId": "my-campaign-123",
      "occurredAt": "2026-06-29T12:34:52Z"
    },
    {
      "eventId": "550e8400-e29b-41d4-a716-446655440002",
      "outboundMessageId": "6ba7b810-9dad-11d1-80b4-00c04fd430c8",
      "eventType": "message-delivered",
      "status": "Sent",
      "deliveryCode": "delivered",
      "providerErrorCode": null,
      "correlationId": "my-campaign-123",
      "occurredAt": "2026-06-29T12:35:10Z"
    }
  ],
  "totalCount": 3,
  "page": 1,
  "pageSize": 50,
  "totalPages": 1
}

Returns the full event timeline for one or more messages. At least one of correlationId or outboundMessageId is required.

Parameter Type Description
correlationId string Filter by correlation ID (batch identifier).
outboundMessageId GUID Filter by a specific message ID.
eventType string Filter by event type (e.g. message-delivered).
status int Filter by message status enum value (0=Ready, 1=Dispatched, 2=Sent, 5=Failed, 6=Unknown).
from ISO date Start of occurred-at range.
to ISO date End of occurred-at range.
page int Page number (default 1).
pageSize int Items per page (default 50, max 500).
sort string Sort order: asc or desc (default desc).

Get Message Summary

GET /api/v1/events/summary?correlationId=my-campaign-123&page=1&pageSize=50
X-Api-Key: <key>

// 200 OK
{
  "items": [
    {
      "outboundMessageId": "6ba7b810-9dad-11d1-80b4-00c04fd430c8",
      "status": "Sent",
      "deliveryCode": "delivered",
      "providerErrorCode": null,
      "occurredAt": "2026-06-29T12:35:10Z"
    },
    {
      "outboundMessageId": "6ba7b810-9dad-11d1-80b4-00c04fd430c9",
      "status": "Failed",
      "deliveryCode": null,
      "providerErrorCode": "http_429",
      "occurredAt": "2026-06-29T12:35:05Z"
    }
  ],
  "totalCount": 2,
  "page": 1,
  "pageSize": 50,
  "totalPages": 1
}

Returns the latest event per unique outbound message ID within a correlation group. Uses DISTINCT ON for efficient one-row-per-message aggregation.

Parameter Type Description
correlationId string Required. The correlation ID to summarize.
page int Page number (default 1).
pageSize int Items per page (default 50, max 500).

Send Bulk Messages

POST /api/v1/messages/bulk-send
X-Api-Key: <key>
Content-Type: application/json

[
  {
    "providerId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "senderAddress": "+15550001111",
    "destinationAddress": "+15559998888",
    "body": "First message"
  },
  {
    "providerId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "senderAddress": "+15550001111",
    "destinationAddress": "+15558887777",
    "body": "Second message"
  }
]

// 202 Accepted — bulk send returns an array of assigned message IDs
[
  "550e8400-e29b-41d4-a716-446655440000",
  "6ba7b810-9dad-11d1-80b4-00c04fd430c8"
]

Messages are grouped by (ProviderId, SenderAddress) and enqueued in Redis in bulk. A single WakeUp is sent per unique provider key.

Redis Workflow

A single message touches six Redis key patterns across its lifecycle (if sent as part of a group). Each pattern serves a distinct purpose in the dispatch → track → poll → cleanup pipeline.

Key Patterns

Pattern Type Purpose
provider-queue:{providerId}:{sender} List FIFO dispatch queue. Messages are RPUSHed by the API and LPOPed by the dispatcher.
pending-receipts-index:{providerId}:{sender} Sorted Set Timeout index. Score = dispatch Unix timestamp. Members are providerMessageIds. Used by pollers to find messages past DispatchedMessageTimeoutInSeconds.
pending-receipts-poll-index:{providerId}:{sender} Sorted Set Poll scheduling index. Score = last poll time. Used by pollers to find messages eligible for the next poll cycle (score ≤ now − StartPollingAfterSeconds).
pending-receipts-data:{providerId}:{sender} Hash Dispatched message payloads. Field = providerMessageId, value = serialized MessageDispatched JSON. Enables webhooks and pollers to resolve context.
tx-provider:{providerMessageId} String Reverse lookup. Maps a providerMessageId back to its providerKey. Enables webhook receipt handlers to find the correct tracking keys.
group-stats:{tenantId}:{correlationId} Hash Real-time aggregated statistics (dispatched, sent, failed, unknown) for a message group. Incremented during dispatch and FlushPipeline persistence.

Lifecycle Phases

Phase 1 — Enqueue

  1. Client calls POST /api/v1/messages/send with a CorrelationId.
  2. API HINCRBY the dispatched count on the group-stats Hash.
  3. API serializes the message and RPUSHes it onto the provider-queue:{providerKey} List.
  4. API publishes a MessageEnqueued event to Redis Stream (events:enqueues) and the Akka EventStream — this signals the EventBatchWriterActor to persist the event with message_status = Ready.
  5. API sends a WakeUp signal to the ProviderQueueCoordinatorActor shard.

Phase 2 — Dequeue & Dispatch

  1. ProviderQueueCoordinatorActor does an atomic LPOP to pull a batch from the dispatch List.
  2. Messages flow through DynamicThrottleActor → MessageWorkerActorBase.
  3. On successful dispatch, the worker calls PublishDispatchedAsync, which writes to four tracking keys and a stream in a single Redis batch:
    • pending-receipts-index Sorted Set — adds providerMessageId with score = current Unix timestamp
    • pending-receipts-poll-index Sorted Set — same entry, used by pollers for scheduling
    • pending-receipts-data Hash — stores the full MessageDispatched payload
    • tx-provider String — stores the serialized ProviderKey, TTL = 3 days
    • provider-wakeup Stream — triggers the PollerManagerActor to start polling

Phase 3 — Delivery Receipts (Webhook or Poll)

  1. Webhook: The external provider POSTs a webhook. The API controller publishes a DeliveryReceiptReceived event to the EventStream.
  2. Poll: The ReceiptPollerActorBase periodically queries pending-receipts-poll-index, reads the payload from pending-receipts-data, fetches status from the provider, and publishes DeliveryReceiptReceived to the EventStream.
  3. Persistence: The EventBatchWriterActor consumes these events and executes the FlushPipeline.
    • It performs the database COPY flush.
    • It HINCRBYs the appropriate sent/failed counters in the group-stats Hash.
    • It calls RemoveFromPollIndexAsync to delete the transaction from pending-receipts-poll-index.
    • Note: the message is intentionally left in pending-receipts-index and pending-receipts-data so subsequent webhooks for the same message can still resolve its context.

Phase 4 — Timeout & Cleanup

  1. The ReceiptPollerActorBase queries pending-receipts-index for messages older than DispatchedMessageTimeoutInSeconds.
  2. For each timed-out message, it publishes a MessageFailed (timeout) event and calls RemoveAsync.
  3. RemoveAsync finally deletes the entry from all four keys (index, poll-index, data, and tx-provider).
  Client                           Redis                              Akka Actors                     Provider API
  ──────                           ─────                              ───────────                     ────────────
     │                                │                                    │                               │
     │  POST /send                    │                                    │                               │
     ├────────────────────────────────┤                                    │                               │
     │  RPUSH provider-queue:{key}    │                                    │                               │
     │  202 { id }                    │                                    │                               │
     │◄───────────────────────────────┤                                    │                               │
     │                                │                                    │                               │
     │                                │  LPOP batch                       │                               │
     │                                │◄───────────────────────────────────┤  ProviderQueueCoordinator     │
     │                                │  [Message, Message, ...]          │                               │
     │                                ├───────────────────────────────────►│                               │
     │                                │                                    │                               │
     │                                │                                    │  HTTP POST /messages          │
     │                                │                                    ├──────────────────────────────►│
     │                                │                                    │◄──────────────────────────────┤
     │                                │                                    │  200 { transactionId }        │
     │                                │                                    │                               │
     │                                │  BATCH:                           │                               │
     │                                │  ZADD pending-receipts-index      │                               │
     │                                │  ZADD pending-receipts-poll-index │                               │
     │                                │  HSET pending-receipts-data       │                               │
     │                                │  SET  tx-provider:{msgId}         │                               │
     │                                │◄───────────────────────────────────┤                               │
     │                                │                                    │                               │
     │                                │  ... time passes ...              │                               │
     │                                │                                    │                               │
     │                                │  SCAN pending-receipts-index:*    │                               │
     │                                │◄───────────────────────────────────┤  PollerManager                │
     │                                │  ZRANGEBYSCORE (timeouts)         │                               │
     │                                │◄───────────────────────────────────┤  ReceiptPollerActor           │
     │                                │  ZRANGEBYSCORE (poll candidates)  │                               │
     │                                │◄───────────────────────────────────┤                               │
     │                                │  HGET (message payloads)          │                               │
     │                                │◄───────────────────────────────────┤                               │
     │                                │                                    │                               │
     │                                │                                    │  GET /receipts?ids=...        │
     │                                │                                    ├──────────────────────────────►│
     │                                │                                    │◄──────────────────────────────┤
     │                                │                                    │  [DeliveryReceipt, ...]       │
     │                                │                                    │                               │
     │                                │  ZREM + HDEL + DEL (cleanup)      │                               │
     │                                │◄───────────────────────────────────┤                               │

Provider Implementations

Provider Worker Actor Receipt Poller Channel
Trumpia TrumpiaWorkerActor TrumpiaReceiptPollerActor SMS/MMS
Twilio TwilioWorkerActor TwilioReceiptPollerActor SMS/MMS
Telnyx TelnyxWorkerActor TelnyxReceiptPollerActor SMS/MMS

Note: The Meta (WhatsApp) provider type and email channel are defined in the data model but are not yet wired into the actor-based dispatch engine. They are currently managed through the admin website CRUD layer only.

Providers are registered in the ProviderFactory during startup and use dynamic pool sizing: poolSize = max(MinWorkersPoolSize, MessagesPerSecond × EstimatedLatencyInSeconds).

Delivery Receipts

The platform supports two receipt collection mechanisms:

1. Webhooks (Push — Preferred)

External providers POST delivery status updates to:

Webhook Endpoint Validation
Twilio POST /api/webhooks/twilio HMAC-SHA1 X-Twilio-Signature validation using the dynamic AuthToken from provider_configs.credentials
Telnyx POST /api/webhooks/telnyx Ed25519 signature validation via telnyx-signature-ed25519 and telnyx-timestamp headers, verified against the PublicKey from provider_configs.credentials
Trumpia POST /api/webhooks/trumpia —

Webhook updates are forwarded to EventBatchWriterActor for batched persistence.

2. Polling (Pull — Fallback)

The PollerManagerActor periodically scans Redis for active provider keys (providers with pending dispatched messages). For each active key, the ProviderPollerCoordinatorActor shard creates a provider-specific ReceiptPollerActor that queries the external API for delivery status.

Configuration:

  • StartPollingAfterSeconds: How long after dispatch before polling begins.
  • DeliveryReceiptPollIntervalInSeconds: How often the scan runs.
  • DispatchedMessageTimeoutInSeconds: Messages without a receipt after this time are marked Unknown.

Dynamic Throttling

The DynamicThrottleActor implements a token bucket algorithm per provider sender:

  • MaxMessagesPerSecond: Configured per sender in provider_senders.max_messages_per_second.
  • HighWatermark × MaxMessagesPerSecond: When in-flight + queued messages exceed this, the throttle sends SlowDown to the coordinator, which pauses batch fetching.
  • LowWatermark × MaxMessagesPerSecond: When both metrics drop below this, the throttle sends ResumeSpeed and fetching resumes.

This creates true end-to-end backpressure — no polling, no busy-waiting.

Project Structure

src/
├── MessagingService.AppHost/          # .NET Aspire app host
├── MessagingService.ServiceDefaults/  # OpenTelemetry, health checks, service defaults
├── MessagingService.Shared/           # Shared DTOs (MessageDto), enums (ProviderType, MessageStatus)
│
├── service/
│   ├── MessagingService.Api/          # HTTP API layer (endpoints, auth, OpenAPI)
│   ├── MessagingService.Contracts/    # Shared models (ProviderKey, ProviderConfig, Message events, repository interfaces)
│   ├── MessagingService.Core/         # Core actor base classes, messages, settings, provider factory
│   ├── MessagingService.Database/     # Repository implementations (PostgreSQL + Redis)
│   ├── MessagingService.Dispatching/  # Dispatch shard: ProviderQueueCoordinatorActor + DynamicThrottleActor
│   ├── MessagingService.Polling/      # Poll shard: PollerManagerActor + ProviderPollerCoordinatorActor
│   │
│   ├── providers/
│   │   ├── MessagingService.Trumpia/  # Trumpia worker + poller actors
│   │   ├── MessagingService.Twilio/   # Twilio worker + poller actors
│   │   └── MessagingService.Telnyx/   # Telnyx worker + poller actors
│   │
│   └── simulator/
│       ├── MessageService.MessageBrokerSimulator/  # Fake provider API for integration testing
│       └── MessagingService.StressSimulator/       # High-load traffic generator for stress testing
│
└── website/
    ├── MessagingService.Website.Api/              # Admin CRUD API (providers, senders, tenants, messages)
    ├── MessagingService.Website.Application/      # Application services
    ├── MessagingService.Website.Application.Contracts/  # DTOs and service interfaces
    ├── MessagingService.Website.Domain/           # EF Core entities (ProviderConfig, ProviderSender, MessageTransaction, etc.)
    ├── MessagingService.Website.Data.EntityFrameworkCore/  # EF Core DbContext, migrations, repositories
    ├── MessagingService.Website.Components/       # Shared Blazor components
    └── MessagingService.Website.Blazor/           # Blazor admin frontend

tests/
└── MessagingService.Tests/            # Unit + integration tests

Database Model

Core Tables

Table Schema Purpose
tenants ms Multi-tenant configuration
tenant_authentication ms Per-tenant API secrets
provider_configs ms Provider credentials and type
provider_senders ms Sender addresses per provider with per-sender rate limits
message_transactions ms Message dispatch log with status tracking
attachments ms Media file metadata linked to messages
message_events ms Append-only message event journal. Records the full timeline of each message — enqueued, dispatched, sent, delivered, failed — with canonical status, delivery code, and provider error code columns for queryability without JSON parsing. Bulk-written via PostgreSQL COPY. Indexed on correlation_id, outbound_message_id, occurred_at, and (tenant_id, occurred_at). Created dynamically at startup.
webhook_notifications ms Webhook notification journal — logs outbound callback attempts (dispatched, delivered, failed) with request/response payloads, bulk-written via PostgreSQL COPY (created dynamically at startup).
message_group_stats_history ms Auto-snapshotted historical data for idle message groups tracked by CorrelationId (created dynamically at startup).

Key Relationships

tenants ──┬── tenant_authentication
          └── message_transactions

provider_configs ──┬── provider_senders
                   └── message_transactions

provider_senders ── message_transactions

message_transactions ── attachments

Full schema details are available in the ms-service.dbml file in the root directory or generated EF Core migrations.

Configuration Reference

Most tuning parameters live under the MessagingServiceSettings key in appsettings.json, while Akka.NET cluster configuration lives under the top-level AkkaSettings key.

Environment Variable Mapping

ASP.NET Core maps colons (:) to double underscores (__) for environment variable overrides:

MessagingServiceSettings__Database__WriterFlushIntervalInSeconds=2
MessagingServiceSettings__Throttler__HighWatermark=5

Database (MessagingServiceSettings:Database)

Parameter Default Description
TimeoutInSeconds 300 Max duration for a single database query.
WriterFlushIntervalInSeconds 5 How often EventBatchWriterActor bulk-flushes in-memory message events (dispatches, receipts) to PostgreSQL.
WebhookJournalFlushIntervalInSeconds 2 How often WebhookJournalWriterActor bulk-flushes webhook notification records to PostgreSQL.
EventBatchWriterMaxPendingBeforeForceFlush 1000 Max pending events (dispatches + receipts + failures) before forcing an immediate flush, regardless of the flush interval.
WebhookJournalMaxPendingBeforeForceFlush 1000 Max pending webhook notification records before forcing an immediate flush.

Throttle Watermarks (MessagingServiceSettings:Throttler)

Multiplied against a sender's max_messages_per_second to derive pause/resume thresholds.

Parameter Default Description
HighWatermark 5 In-flight + queued exceeds MaxMPS × HighWatermark → producers pause.
LowWatermark 2 Both counts drop below MaxMPS × LowWatermark → producers resume.

Lifecycle (MessagingServiceSettings:Lifecycle)

Important: DispatchedMessageTimeoutInSeconds must always be greater than both StartPollingAfterSeconds and DeliveryReceiptPollIntervalInSeconds. Otherwise messages time out before the poller can recover their status.

Parameter Default Description
ProviderMaxIdleTimeInSeconds 1800 Shuts down a provider's entire actor tree after this many seconds of inactivity.
DispatchedMessageTimeoutInSeconds 540 A Dispatched message with no receipt after this many seconds is marked Unknown.
DispatchedMessageCacheTtlInSeconds 259200 The Time-To-Live (TTL) in seconds for dispatched message cache entries in Redis. Must be greater than timeout.
StartPollingAfterSeconds 100 Delay after dispatch before the poller actively queries the provider API.

Circuit Breaker (MessagingServiceSettings:CircuitBreaker)

Applied to both worker and poller actors per provider.

Parameter Default Description
MaxFailures 5 Consecutive failures to trip the breaker.
CallTimeoutInSeconds 10 Max duration for a single HTTP call.
ResetTimeoutInSeconds 60 How long the breaker stays open before testing recovery.

Provider Integration (MessagingServiceSettings:ProviderIntegration)

Parameter Default Description
EstimatedLatencyInSeconds 2 Estimated round-trip latency, used for dynamic worker pool sizing.
MinWorkersPoolSize 5 Minimum worker actor pool size per provider.
DeliveryReceiptPollIntervalInSeconds 120 How often PollerManagerActor scans Redis for active providers.
DeliveryReceiptPollBatchSize 500 Number of message IDs per poll request to the external provider.
DeliveryReceiptPollMaxDegreeOfParallelism 10 Max concurrent HTTP connections for receipt polling.

Group Stats (MessagingServiceSettings:GroupStats)

Settings controlling group stats tracking via Redis HINCRBY counters and PostgreSQL auto-snapshots.

Parameter Default Description
TtlHours (required) The TTL in hours for the Redis key storing group statistics. Must be configured explicitly.
SnapshotAfterIdleHours 3 The number of hours a group can remain idle before it is auto-snapshotted to PostgreSQL.
SnapshotScanIntervalHours 1 The interval in hours at which the background snapshot actor scans Redis for idle groups.
WebhookThrottleSeconds 30 The minimum interval in seconds between webhook notifications for the same group (at-most-once throttling).

Service Endpoints (MessagingServiceSettings)

Parameter Description
MediaBaseUrl Base URL used for resolving linked media attachments in outbound MMS messages.
WebhooksBaseUrl Base URL used for constructing incoming webhook notification callbacks.

Metrics (MessagingServiceSettings:Metrics)

Parameter Default Description
PublishIntervalInSeconds 2 How often the metrics collector publishes snapshots to the Akka EventStream.

Akka Cluster (AkkaSettings)

This configuration section is bound to the AkkaSettings top-level key in appsettings.json.

Parameter Default Description
HostName (dynamic) Public hostname/advertised address for cluster discovery. Defaults to the local IPv4 address.
RemotePort 14884 Akka.Remote TCP port.
ManagementPort 18558 Akka.Management HTTP port.
Roles api,dispatcher,poller Comma-separated list of node roles (api, dispatcher, poller).
DiscoveryServiceName messaging-service-discovery Service name for DNS or config-based cluster bootstrap.
ConfigDiscoveryEndpoints [] List of endpoints used for static service discovery (e.g., local development).
RequiredContactPointsNr 2 Number of nodes that must be discovered before the cluster is fully formed. Set to 1 for local single-node development.

Deployment

The service is designed for containerized deployment (Docker / Kubernetes). Nodes declare their roles via the AkkaSettings:Roles environment variable and join an Akka.NET cluster via bootstrap discovery.

Minimal Node Types

Node Roles Description
API Gateway api Public HTTP endpoint — enqueues messages, receives webhooks.
Dispatcher dispatcher Picks up messages from Redis and dispatches them through the throttle/worker pipeline.
Poller poller Scans for pending delivery receipts and polls provider APIs.

A combined node with api,dispatcher,poller works for single-instance deployments.

Dependencies

  • PostgreSQL — Message event journal, provider configs, sender addresses, message transaction log.
  • Redis — Per-provider message queues, dispatched message tracking index.
  • Akka.NET Cluster — Service discovery and shard coordination between nodes.