When the Webhook Wakes the Agent: Reliable Event-Driven AI Workflows

A user-initiated chat is an easy starting point for an agent. A webhook is where production reality begins.

External systems retry deliveries. Events arrive twice, arrive late, arrive out of order, or arrive after a timeout has already caused your worker to retry. A payment may be committed while your process is restarting. A human may approve a step days after the workflow began. If the event wakes an agent that can send messages, update records, provision resources, or spend money, the handler needs more than a model call.

The reliable design is not “receive webhook, ask the LLM, perform action.” It is:

Authenticate the event, persist the fact, deduplicate it, reconcile current state, then let a bounded agent propose work to a durable orchestrator.

The real delivery contract is at-least-once

Webhook providers generally optimize for eventual delivery, not exactly-once processing. Stripe documents automatic retries for up to three days in live mode and explicitly warns that event order is not guaranteed. Shopify documents retries and recommends delivery-ID deduplication, quick acknowledgement, queueing, and reconciliation. 1

That means your system must assume all of the following can happen:

Condition	What it breaks
Duplicate delivery	A charge, email, or record update can happen twice
Out-of-order delivery	A “completed” event can arrive before “created”
Delayed delivery	The payload describes state that is no longer current
Timeout after commit	The provider retries even though the side effect succeeded
Consumer crash before ACK	The same event is processed again
Multiple subscriptions	One business action creates different delivery IDs
Malformed or forged payload	An attacker can trigger an agent if authenticity is skipped
Long human wait	In-process state disappears while approval is pending

Exactly-once delivery is not a promise you can safely build around. Exactly-once business effect is the property you need, and it must be implemented at the boundary where side effects occur.

Keep the webhook endpoint boring

The HTTP endpoint should verify authenticity, validate the envelope, persist the event, enqueue work, and acknowledge quickly. It should not run a long LLM call or wait for a downstream API.

Provider-specific signature rules vary, but the ordering is consistent:

1. Read the raw request body.

2. Verify the provider signature with the official library or documented algorithm.

3. Validate timestamp or replay-window rules where supported.

4. Parse the envelope with a runtime schema.

5. Store the provider, event ID, delivery ID, event type, received time, and payload hash.

6. Enqueue or signal the durable workflow.

7. Return success after durable acceptance—not after the agent finishes.

Shopify’s documentation is explicit that HMAC verification must use the raw body and happen before trusting payload contents. It also distinguishes a delivery ID, useful for deduplicating a specific delivery, from an event ID, useful for correlating deliveries caused by the same merchant action. 2

A minimal TypeScript boundary might look like this:

import crypto from "node:crypto";
import { z } from "zod";

const Envelope = z.object({
id: z.string().min(1),
type: z.string().min(1),
created_at: z.string().datetime(),
data: z.record(z.unknown())
});

function validHmac(rawBody: Buffer, received: string, secret: string): boolean {
const expected = crypto.createHmac("sha256", secret).update(rawBody).digest("base64");
const a = Buffer.from(received, "base64");
const b = Buffer.from(expected, "base64");
return a.length === b.length && crypto.timingSafeEqual(a, b);
}

export async function webhook(req: Request): Promise<Response> {
const raw = Buffer.from(await req.arrayBuffer());
const signature = req.headers.get("x-provider-signature");

if (!signature || !validHmac(raw, signature, process.env.WEBHOOK_SECRET!)) {
return new Response("invalid signature", { status: 401 });
}

const parsed = Envelope.safeParse(JSON.parse(raw.toString("utf8")));
if (!parsed.success) return new Response("invalid event", { status: 400 });

await eventStore.accept({
provider: "provider-name",
eventId: parsed.data.id,
eventType: parsed.data.type,
payload: parsed.data,
payloadSha256: crypto.createHash("sha256").update(raw).digest("hex")
});

await workflow.signalWithStart({
workflowId: `account-${accountId(parsed.data)}`,
signal: "externalEvent",
payload: parsed.data
});

return new Response(null, { status: 200 });
}

The example intentionally treats the model as downstream of event acceptance. If the agent provider is unavailable, the webhook should still be accepted and recoverable.

Deduplicate deliveries with a durable inbox

A Redis lock with a short TTL is not enough when the provider can retry for days or when a worker crashes after the side effect but before deleting the lock. Use a durable inbox table with a uniqueness constraint, and make event acceptance transactional.

CREATE TABLE inbound_events (
provider              text        NOT NULL,
delivery_id           text        NOT NULL,
event_id              text        NOT NULL,
event_type            text        NOT NULL,
payload               jsonb       NOT NULL,
payload_sha256        text        NOT NULL,
status                text        NOT NULL DEFAULT 'accepted',
first_seen_at         timestamptz NOT NULL DEFAULT now(),
processed_at          timestamptz,
attempts              integer     NOT NULL DEFAULT 0,
last_error            text,
PRIMARY KEY (provider, delivery_id)
);

CREATE UNIQUE INDEX inbound_events_event_identity
ON inbound_events (provider, event_id, event_type);

Which identity belongs in the uniqueness constraint depends on the provider. If two subscriptions legitimately deliver the same event to different workflows, delivery identity and business-event identity are not interchangeable. Store both. Decide whether you deduplicate per delivery, per business event, or per side effect.

The inbox is the first half of the reliable pattern. The second half is an outbox or durable signal that connects the accepted fact to processing. Do not acknowledge an event before it is durably recorded, and do not rely on a process-local queue that disappears on restart.

Separate event facts from agent intentions

An event says what an external system reported. It does not tell the agent to perform an arbitrary action.

Use a deterministic normalization step to turn a provider payload into a typed internal fact. Then let the agent produce a structured intention. The orchestrator validates that intention against policy before any activity can mutate real state.

{
"event": {
"provider": "shopify",
"event_id": "evt_123",
"delivery_id": "del_456",
"type": "orders/paid",
"occurred_at": "2026-10-14T08:30:00Z"
},
"intention": {
"kind": "prepare_fulfillment_review",
"order_id": "order_789",
"reason": "payment_confirmed",
"requested_effects": ["create_internal_task"],
"confidence": 0.91
}
}

The agent should not receive a raw webhook as an instruction channel. Payload fields can contain customer text, document content, or attacker-controlled strings. Treat them as data. The system prompt should make the distinction clear, but server-side authorization and effect policy must enforce it.

This separation is consistent with the ESAA event-sourcing research architecture, which keeps LLM cognition separate from deterministic execution. The agent emits structured intentions and proposed changes; an orchestrator validates, appends events, and projects state. An append-only event log enables replay, auditability, and correction without rewriting history. 3

Use a durable recipient, not a sleeping process

After the webhook is accepted, the workflow may need to wait for another event, an external job, or human approval. A process-local await, cron timer, or in-memory promise is the wrong place to hold that state.

Temporal’s delayed-callback pattern uses Signal-with-Start for inbound webhooks. The handler calls the durable recipient and can return immediately. The workflow is started if necessary and receives the signal atomically, avoiding a race between checking whether a workflow exists and delivering the event. Temporal also provides durable timers and asynchronous activity completion for external callbacks. 4

A conceptual Python workflow can keep the event loop deterministic:

from datetime import timedelta
from temporalio import workflow

@workflow.defn
class OrderAgentWorkflow:
def __init__(self):
self.events = []
self.approved = False

@workflow.signal
async def external_event(self, event: dict):
self.events.append(event)

@workflow.signal
async def approve(self, reviewer: str):
self.approved = True
self.events.append({"kind": "approved", "reviewer": reviewer})

@workflow.run
async def run(self, order_id: str):
await workflow.wait_condition(lambda: self.events)
normalized = await workflow.execute_activity(
normalize_event,
self.events.pop(0),
start_to_close_timeout=timedelta(seconds=30),
)

intention = await workflow.execute_activity(
agent_propose_next_step,
normalized,
start_to_close_timeout=timedelta(minutes=5),
)

if intention["requires_review"]:
await workflow.wait_condition(
lambda: self.approved,
timeout=timedelta(days=7),
)

return await workflow.execute_activity(
apply_bounded_effect,
intention,
start_to_close_timeout=timedelta(minutes=2),
)

The exact framework can vary. The design invariant is that the orchestrator owns durable state, waits, retries, and effect permissions. The agent performs reasoning inside an activity or worker boundary and cannot turn an external payload directly into an unreviewed write.

Handle ordering explicitly

Do not assume event timestamps provide a total order. Stripe explicitly says event delivery order is not guaranteed and recommends retrieving current objects when needed. 1

For each event, record:

• Provider event time: when the source says the event occurred.
• Delivery time: when your endpoint received it.
• Sequence or version: if the provider supplies one.
• Observed resource version: the version you fetched during processing.
• Workflow revision: the state version your orchestrator accepted.

When a new event arrives, choose one of three policies:

1. Commutative update: apply it regardless of order because the final projection is safe.
2. Version-gated update: apply only if the event version is newer than the stored version.
3. Reconciliation: retrieve the current source object and derive the next action from present state rather than trusting a stale payload.

Agents are especially vulnerable to stale events because a model can write a confident explanation around obsolete data. Pass the normalized event and the current authoritative state separately, with provenance labels.

Make every side effect idempotent

A workflow retry is not a duplicate only at the webhook level. The LLM activity can time out after generating a valid proposal. The outbound API can commit and then lose its response. The worker can crash after an email is accepted but before recording success.

Temporal describes activity execution as at least once by default and recommends making activities idempotent or ensuring repeated execution has no unwanted side effect. It gives idempotency-key patterns based on a business key or workflow/activity identity, and shows a uniqueness constraint in an operations table as a practical guard. 5

Use a side-effect ledger:

CREATE TABLE effect_ledger (
idempotency_key text PRIMARY KEY,
workflow_id     text NOT NULL,
effect_kind     text NOT NULL,
target_id       text NOT NULL,
request_hash    text NOT NULL,
status          text NOT NULL,
result          jsonb,
created_at      timestamptz NOT NULL DEFAULT now(),
completed_at    timestamptz
);

The activity should claim the key and perform the effect in a transaction where possible. If the key already exists with a completed result, return that result. If it exists as in-progress, follow a defined recovery path instead of blindly performing the effect again.

async function applyEffect(input: EffectInput) {
const key = `${input.workflowId}:${input.stepId}`;
const requestHash = hashCanonical(input.payload);

const claimed = await db.tx(async tx => {
const row = await tx.oneOrNone(
`INSERT INTO effect_ledger
(idempotency_key, workflow_id, effect_kind, target_id, request_hash, status)
VALUES ($1, $2, $3, $4, $5, 'started')
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING *`,
[key, input.workflowId, input.kind, input.targetId, requestHash]
);
if (!row) return tx.one("SELECT * FROM effect_ledger WHERE idempotency_key = $1", [key]);
return row;
});

if (claimed.status === "completed") return claimed.result;
if (claimed.request_hash !== requestHash) throw new Error("idempotency key reused with different payload");

const result = await provider.execute(input.payload, { idempotencyKey: key });
await db.none(
`UPDATE effect_ledger SET status='completed', result=$2, completed_at=now()
WHERE idempotency_key=$1`,
[key, result]
);
return result;
}

The provider’s own idempotency support is valuable, but keep a local ledger for audit, reconciliation, and systems that do not offer keys.

DLQs are not recovery plans

A dead-letter queue is useful when an event cannot be processed after bounded retries. It is not the end of the design.

Each DLQ record should retain the original event identity, payload hash, failure class, attempts, first and last failure times, workflow ID, and whether any side effect may have committed. Operators need to distinguish:

• Permanent invalid input: quarantine and alert; do not retry forever.
• Authentication failure: investigate secret rotation or provider configuration.
• Transient dependency failure: retry with backoff and jitter.
• Stale event: reconcile current state, then close or supersede the event.
• Unknown effect outcome: query the provider or effect ledger before retrying.
• Agent quality failure: preserve the trace, route for review, and add a regression case.

Shopify recommends a periodic reconciliation job that retrieves missed data through APIs. 2 That is the key complement to webhook delivery: a webhook is a trigger and a hint, not the only source of truth.

reconciliation:
cadence: "*/15 * * * *"
sources:
- provider: shopify
resource: orders
lookback: 2h
cursor: updated_at
actions:
- compare_provider_state_to_projection
- enqueue_missing_events
- mark_stale_workflows
- alert_on_unresolvable_divergence

Reconciliation should be safe to run repeatedly. It should create the same normalized fact and idempotent effect key rather than bypassing the normal processing path.

Observe the event-to-effect chain

A request ID at the HTTP edge is not enough. Propagate a correlation ID through the entire event lifecycle:

provider delivery
-> inbound event row
-> queue or durable signal
-> workflow execution
-> agent activity
-> intention event
-> policy decision
-> effect ledger
-> provider side effect
-> reconciliation result

Useful metrics include:

• signature failures and replay-window rejects;
• duplicate delivery rate by provider and event type;
• time from receipt to durable acceptance;
• queue age and workflow wait age;
• out-of-order and stale-event rate;
• agent proposals rejected by policy;
• effect-ledger conflicts and unknown outcomes;
• DLQ volume by failure class;
• reconciliation divergence and repair time;
• completed business outcomes, not just successful model calls.

Log the event ID, delivery ID, workflow ID, agent run ID, effect key, policy version, and source version. Redact payload contents according to tenant and privacy policy. An event-driven agent is only debuggable if you can answer what arrived, what the model proposed, what the policy allowed, what actually changed, and whether the source later agreed.

References & Community Insights

Production checklist

1. Is the raw webhook authenticated before payload parsing and agent use?

2. Is the event durably recorded before returning HTTP success?

3. Are delivery identity and business-event identity stored separately?

4. Can duplicate, late, and out-of-order events be processed without duplicate effects?
5. Does the agent emit a structured intention rather than directly holding write credentials?

6. Are workflows durable across restarts and human waits?

7. Does every side effect have an idempotency key and an effect ledger?

8. Is there a DLQ with replay and operator ownership?

9. Is reconciliation able to recover missed or stale webhook state?

10. Can one correlation ID connect the delivery, workflow, agent decision, policy result, and external effect?

Event-driven agents are powerful because they turn systems into active participants: a payment, ticket, document, deployment, or approval can wake a workflow without a human opening a chat. But the trigger is only the beginning. Production reliability comes from treating every event as untrusted, duplicated, delayed input; every model output as a proposal; and every external write as a durable, idempotent, auditable effect.