Reference architecture: Real-time data for agentic workflows

The systems that need real-time data usually share the same shape: people, devices, software, and business rules are all acting on operational state while that state is still changing. Dispatch boards, field-service apps, fleet tracking, multiplayer experiences, support queues, IoT monitoring, and fraud response systems all depend on signals that can lose value quickly: who is connected, where an asset is, which task is active, whether a device has timed out, or whether a human approver is available.

Agentic workflows belong in this same class. An agent that plans or acts against stale state can make a technically valid decision that is wrong for the current moment. The point of real-time architecture is not to stream every event by default; it is to reduce the gap between the world an agent reasons about and the world it acts on.

Climate LLC, the digital farming arm of Bayer Crop Science, is a useful example. It uses PubNub to power real-time updates for its FieldView platform, including Remote View workflows where farm managers can monitor equipment usage from outside the field. PubNub Presence helps the platform track equipment usage and push device-status changes as they happen, so teams can coordinate around current field conditions instead of waiting for a delayed batch view. Agentic workflows need the same discipline: fresh operational signals where they change a decision, reliable delivery to the right subscribers, and replayable history for verification and audit.

You're wasting budget and adding operational risk by defaulting to streaming for entire agentic systems. The faster fix is a per-step freshness map and a hybrid architecture that applies streaming only where it changes decisions or safety outcomes.

Workflow overview: the agentic loop and why freshness matters

Agentic workflows follow a repeatable loop: perception → planning/reasoning → action → observation/feedback → audit/verifier. Freshness matters differently at each step:

• Perception (telemetry, user input, system events): Agents need recent state to form accurate situational awareness.
• Planning/Reasoning (LLM or policy engine): Decisions depend on context; stale context can produce incorrect or unsafe actions.
• Action (actuator commands, API calls): Commands must target the intended state - sometimes the difference between "on" and "off" is seconds.
• Verifier/Acceptance (human or automated gate): Verifiers require sufficient context and provenance to accept or block an action.
• Audit/Replay (post-hoc review): Requires full history and reliable timestamps for deterministic replay and compliance.

Agentic systems are brittle when a single, undifferentiated "real-time" decision is applied everywhere. The correct approach: define where low-latency state materially changes the agent's choice or where safety/regulation requires immediate verification.

Mapping freshness to workflow steps

Use coarse "freshness bands" as a decision tool, not hard rules. The bands below are practical starting points and should be validated against your SLAs, regulatory needs, and SME input.

• Band A - sub-second (ms): perception used directly in fast-loop reasoning (high-frequency telemetry for collision avoidance, live operator presence).
• Band B - seconds: short-lived context for decision-making (ongoing human session state, recent customer interactions).
• Band C - minutes: operational state used for planning but tolerant to short delays (batch-updated inventory counts, recent analytics).
• Band D - hours+: archival context for model training, long-term audit, or analytics.

Decision rules: when to stream vs API/batch

Use a short scoring checklist. Stream if any of these apply:

• Decision Sensitivity: The agent's choice would change within the freshness band (e.g., perception feed used in the same reasoning cycle).
• Safety or Compliance: Safety-critical actions require low-latency context or human-in-the-loop gating.
• Subscriber Fan-out: Multiple interested real-time consumers (dashboards, live agents, operators) need the event promptly.
• User Experience SLA: Users expect live updates (collaboration, presence indicators).
• Operational Cost Justified: The value of reduced latency exceeds the cost of always-on streaming.

If none apply, prefer API calls or batched events into a message queue or data lake and serve periodic updates.

Examples:

• Telemetry for collision avoidance: stream (Band A).
• Weekly model training features: batch (Band D).
• Approval workflow where verifier must see the momentary state: stream to verifier channel; persist for replay.

Verifier and acceptance patterns: where to put checks

Common verifier placements:

• Pre-action gate: verifier inspects proposed action plus a snapshot before the action is taken. Use when preventing unsafe actions is critical.
• Post-action acceptance: action executes immediately and verifier can revert or remediate on rejection. Use if latency constraints require immediate action and system tolerates rollbacks.
• Parallel observer: verifier receives a copy and flags anomalies for human review later. Useful for monitoring and continuous improvement.

Design considerations:

• Provide verifiers a contextual snapshot: not just the action but the recent event history. Storage/playback is essential to reconstruct the state at decision time.
• Keep verifier tokens least-privilege and short-lived. Grant/revoke frameworks prevent lateral access to tenant streams.
• Use deterministic replay (timestamps, trace IDs) to support debugging and regulatory audits.

PubNub features such as storage/playback and per-message timestamps support verifier needs; see replay use cases here.

Storage, replay, and auditability

Auditability requires:

• Immutable event history with reliable timestamps and traceability to the source system.
• Fast query and playback of event windows surrounding decisions (e.g., "show the 30 seconds before the action").
• Correlation IDs linking API calls, persisted state, agent reasoning logs, and streamed messages.

Patterns:

• Dual-write: persist canonical record before publishing to the stream. This ensures replay matches the persisted state.
• Event envelopes: include metadata (trace_id, timestamp, origin, schema version) in each published message.
• Deterministic replay: support replay to dev/test systems and verifiers with the same processing logic used in production.

Mapping to PubNub-style building blocks

Use these building blocks for the streaming slices of your architecture:

• Publish/Subscribe: low-latency fan-out to agents, UIs, and verifiers.
• Channels: map to object/tenant/workflow identities (channel naming by asset/tenant is an established pattern; see Real-Time Asset Tracking for Fleets.
• Presence: operator and agent connectivity state for live coordination.
• Storage / Playback: replay recent windows and support verifier snapshots.
• Stream Controller: edge routing and transformation - filter sensitive fields before forwarding to agents (Data Communication Interface).
• Access Control: grant/revoke and short-lived tokens to enforce least privilege.
• SDKs & Multiplexing: single socket, multiple channels for efficient client connections.
• Global Data Stream Network: geo-routing, replication, and resilience for globally distributed agents.

Channel naming suggestions:

• tenant:{tenant_id}:workflow:{workflow_id}:object:{object_id}
• verifier:{workflow_id}:human
• audit:{tenant_id}

Keep channels narrow to limit blast radius and reduce message parsing on clients.

Operational concerns and failure modes

• Latency SLAs: define SLOs per band and measure end-to-end (API → persistence → publish → subscriber process).
• Multiplexing: prefer SDKs that manage socket lifecycle and channel subscriptions to avoid client reconnection storms.
• Tokenization: issue short-lived tokens bound to channel scopes and revoke on suspect behavior.
• Retries & Idempotency: design subscribers to handle duplicate delivery and ensure idempotent side-effects.
• Backpressure: provide graceful degradation - drop non-critical streams, switch to summarized updates, or fall back to pull API.
• Testing: include synthetic replay, chaos testing of network partitions, verifier rejection scenarios, and load testing for expected fan-out.

Short checklist for engineers

• Map each workflow step to a freshness band (A-D). Validate with SMEs.
• Implement dual-write: persist canonical state, then publish event.
• Name channels by tenant/object/workflow and minimize channel breadth.
• Route sensitive fields through Stream Controller for redaction/enrichment.
• Create verifier channels with least-privilege tokens and deterministic replay access.
• Enable storage/playback and include trace IDs + timestamps in every event.
• Define SLOs for each freshness band and create monitoring/alerts.
• Pilot: pick one workflow, stream only the steps that score high for Decision Sensitivity or Safety.

Download the reference architecture workflow or talk to us about your real-time agent workflow.