DOCS
Ask AI

Message aggregation

Applications that design channels around individual users or sessions, one channel per conversation or per device, eventually need to aggregate what's happening across all of them. Logging, analytics, moderation, and feeding a downstream system all need that view. Architectural choices introduces the shape of the fix: a small, fixed set of ingress channels that a server subscribes to, instead of one subscription per user channel. This page works through why the direct approach stops scaling and how the hashing that spreads traffic across that fixed set actually works.

Why subscribing to every channel doesn't scale​

The direct approach is also the first one most teams reach for. Design channels around user interactions, then have a backend server subscribe to each one as it's created, so it sees every message that matters. It works while the channel count is small, and it fails in specific, predictable ways once it isn't:

  • The subscription list grows without bound, tracking your channel count rather than any fixed capacity.
  • Every new channel needs its own added subscription, and adding it without a race against that channel's first publish takes its own coordination.
  • Load on the subscribing server is a function of how your users behave, not a number you set, so it's hard to provision for.
  • Traffic across channels is rarely even. A few very active channels can concentrate load on whatever subscribed to them.
  • None of the above gets easier by adding servers, because the thing that needs to scale, the subscription list, isn't partitioned across them.

The fix isn't a bigger server. It's decoupling the number of things you subscribe to from the number of channels your users create.

Shard publishes across a fixed set of ingress channels​

Instead of one channel per user feeding a single aggregator, publish into a small, fixed set of ingress channels chosen by a consistent hash. Run one subscriber process per ingress channel. The set size doesn't change as your user base grows, so neither does your subscription list.

Each client hashes its stable ID, reduces the result with a modulo of the shard count, and publishes to the ingress channel that matches. The one subscriber process attached to that ingress channel then receives the message. Several clients can land on the same ingress channel.


A consistent hash is what keeps that distribution even instead of accidental. Hash a stable identifier the client already has, such as a user or session ID, then reduce it to a shard number with a modulo:

const shardCount = 3;
const shard = hash(clientId) % shardCount; // hash from a library such as crc32 or fnv-1a
const ingressChannel = `server-inbound-${shard}`;

CRC32 is a common choice for the hash function itself, the same one several distributed caching systems use to spread keys across nodes. FNV-1a works as well. Neither is built into a language runtime, so pick a small library implementation for whichever one you choose.

What matters more than the specific algorithm is hashing a value that's stable per client. The same client's traffic must always land on the same shard. Hashing something that changes between messages instead scatters one client's traffic across every shard, defeating the point. A poor choice of hash input, or too few shards for your traffic, can still concentrate load on one shard and recreate the exact bottleneck sharding was meant to avoid.

Match subscriber processes to shards, not the other way around​

Run one subscriber process per ingress channel, and size the ingress set to match, commonly one channel per CPU core on the machine that aggregates them. Use a consistent naming scheme such as server-inbound-0, server-inbound-1, and so on, so the shard number from the hash maps directly to a channel name. The operating system aligns one process per core efficiently. Running more subscriber processes than cores adds contention instead of throughput. Growing capacity means adding shards and matching processes, not stacking additional processes onto the same fixed set.

This is a different sharding than an audience-facing channel needs​

"Sharding" describes two different problems that both show up in a PubNub application, and the fix for one doesn't address the other. The pattern on this page shards on the ingress side: many publishers feed a small, fixed set of channels so a server-side aggregator's subscription count stays flat. Rate limiting shards on the audience side instead. It splits the subscribers of one very large channel into several smaller ones, such as by region or language, so each subscriber receives less traffic. Confirm which side of the traffic you're trying to shape before reaching for either.

Next steps​

  • Architectural choices. Where this fan-in pattern fits among the rest of your channel design decisions.
  • Rate limiting. Keep a single high-occupancy channel usable instead of aggregating many channels into one.
  • Pub/Sub. The channel and subscription model this pattern builds on.
  • Data storage. Read aggregated data back on your own schedule with Message Persistence instead of processing everything live.
  • Send messages effectively. The publish-side checklist for a channel or ingress set under heavy load.

Was this page useful?

Last updated on