Idempotency Is a Product Feature
Your users do not care that their Wi-Fi dropped between your API receiving a request and your server returning a response.
They care that clicking "Pay" once did not charge them twice.
They care that refreshing the page did not create two orders.
They care that a mobile app retrying after a tunnel ride did not send duplicate invite emails to the whole company.
That is why idempotency is not only distributed-systems hygiene. It is product behavior.
Most teams meet idempotency during payment integration.
Stripe asks for Idempotency-Key.
AWS asks for ClientToken.
You add a UUID header, cache the response, and move on.
But that framing is too small.
Idempotency defines whether your product remains predictable when the network stops being perfect.
And networks are never perfect.
Requests time out.
Mobile apps get killed mid-flight.
Browsers retry.
Load balancers retry.
SDKs retry.
Users double-click.
Queue consumers receive the same message again.
Webhooks arrive multiple times.
Deployments interrupt requests after the database commit but before the response.
None of these are exotic failures.
They are normal operating conditions for systems that communicate over networks.
If your product cannot answer:
"What happens when the same intent arrives twice?"
then your product has a gap.
The failure users see is not:
"Non-idempotent POST request."
They see:
Duplicate charge
Duplicate booking
Two cloud machines running
Ten identical Slack alerts
Account created but onboarding failed
"Something went wrong" despite the action succeeding
Support unable to determine whether an order exists
That last one matters more than engineers often admit.
Ambiguity is product debt.
The most expensive state in distributed systems is not failure.
It is:
"We do not know."
The Request That Disappeared
Consider a checkout flow:
User clicks Pay.
Client sends
POST /payments.Server creates the payment successfully.
Response gets lost due to a network timeout.
Client does not know whether the payment happened.
User clicks Pay again.
What should happen?
Without idempotency, the second request may create a second payment.
// Without idempotency - double charge
const res = await fetch('/payments', {
method: 'POST',
body: JSON.stringify({ amount: 4999, currency: 'usd' }),
});
if (!res.ok) {
// Timeout or network error - user retries
const retry = await fetch('/payments', {
method: 'POST',
body: JSON.stringify({ amount: 4999, currency: 'usd' }),
});
// Two charges. Refund workflow. Support ticket. Lost trust.
}
You now have:
Duplicate charge
Refund workflow
Support ticket
Lost trust
Finance reconciliation work
User wondering whether your product is safe
With idempotency:
// With idempotency - one charge
const key = crypto.randomUUID(); // Generated once, on user action
const sendPayment = async (retryKey: string) => {
const res = await fetch('/payments', {
method: 'POST',
headers: { 'Idempotency-Key': retryKey },
body: JSON.stringify({ amount: 4999, currency: 'usd' }),
});
return res.json();
};
// First attempt
await sendPayment(key);
// On timeout, retry with SAME key
await sendPayment(key);
// Server recognizes key, returns original result.
// One payment. One receipt. One coherent story.
Notice what changed.
Not just database correctness.
The user experience changed from:
"Did you charge me twice?"
to
"The network was flaky, but my payment went through."
That is product quality.
Retries Are Product Behavior
We often treat retries as infrastructure plumbing.
Exponential backoff
Jitter
Timeout tuning
Circuit breakers
All useful.
None answer the real question.
Can the caller retry safely?
A retry policy without idempotency is a machine gun attached to your foot.
If an operation:
creates a resource
sends money
reserves inventory
issues a token
dispatches notifications
triggers third-party side effects
then retries can multiply outcomes.
// Retry without idempotency - dangerous
async function deployCluster(config: ClusterConfig) {
try {
return await api.post('/clusters', config);
} catch (err) {
if (isTimeout(err)) {
// Retry - but the first call may have succeeded
return await api.post('/clusters', config); // Two clusters now running
}
throw err;
}
}
A reliable system needs two promises:
Temporary failure should be recoverable.
Retrying must not create additional user-visible side effects.
Idempotency is the contract that makes both possible.
// Retry with idempotency - safe
async function deployCluster(config: ClusterConfig) {
const key = getDeploymentKey(config); // Derived from user action, not request
try {
return await api.post('/clusters', config, {
headers: { 'Idempotency-Key': key },
});
} catch (err) {
if (isTimeout(err)) {
// Retry with same key - server returns existing cluster or processes
return await api.post('/clusters', config, {
headers: { 'Idempotency-Key': key },
});
}
throw err;
}
}
Same Payload Does Not Mean Same Intent
A common implementation hashes the request body:
hash(userId + requestBody)
It feels clever.
It is often wrong.
Imagine a user launches two identical EC2 instances.
Same:
Image
Region
Instance type
Payload
Are they duplicates?
Maybe.
Maybe not.
The payload cannot reliably tell you intent.
// Payload-based dedup - flawed
function generateKey(req: Request) {
const body = req.body;
return `${body.userId}-${body.image}-${body.region}-${body.instanceType}`;
}
// User launches two identical instances intentionally
// Both get same key - second one rejected
// User confused: "Why won't my second instance start?"
Instead, let the client generate an explicit idempotency key.
The client is saying:
These retries belong to one logical action.
// Client-controlled key - correct
function onDeployClick() {
const key = crypto.randomUUID(); // New key per user action
api.post('/instances', payload, {
headers: { 'Idempotency-Key': key },
});
}
// Same key + same caller = same operation (retry)
// New key + identical payload = possibly new operation (intentional)
That subtle distinction protects product semantics.
Good Idempotency Isn't "Ignore Duplicates"
Stripe compares request parameters when the same key is reused.
Suppose:
checkout-123
first means:
Buy Pro Plan
Later the same key is used for:
Buy Enterprise Plan
Returning the old response silently would hide a bug.
Instead:
Reject the request.
async function handleIdempotentRequest(key: string, payload: Payload) {
const existing = await db.idempotencyKey.findUnique({ where: { key } });
if (existing) {
// Same key reused - check if payload matches
if (existing.requestHash !== hashPayload(payload)) {
return new Response(
JSON.stringify({ error: 'Idempotency key reused with different parameters' }),
{ status: 422 }
);
}
// Same key, same payload - return stored response
return new Response(existing.responseBody, { status: existing.responseStatus });
}
// First request - process and store
const result = await processPayment(payload);
await db.idempotencyKey.create({
data: {
key,
requestHash: hashPayload(payload),
responseBody: JSON.stringify(result),
responseStatus: 200,
resourceId: result.id,
},
});
return new Response(JSON.stringify(result), { status: 201 });
}
Good idempotency means:
Honor one declared intent consistently.
Idempotency Needs Product Design
Adding an Idempotency-Key header is necessary.
It is not sufficient.
Suppose a payment provider times out.
You do not know whether the charge succeeded.
Do not show:
Payment failed.
Do not encourage another payment attempt.
Instead say:
Your payment is processing. Please do not retry. We'll confirm the status shortly.
That message reflects backend truth.
A healthy product state model looks like:
Processing
Succeeded
Failed
Requires Action
Reconciling
Reversed
// Product state machine for uncertain outcomes
type PaymentState =
| { status: 'processing'; message: 'Your payment is being processed.' }
| { status: 'succeeded'; receiptUrl: string }
| { status: 'failed'; reason: string }
| { status: 'requires_action'; redirectUrl: string }
| { status: 'reconciling'; message: 'Checking with payment provider...' };
function getPaymentDisplay(state: PaymentState) {
switch (state.status) {
case 'processing':
return {
title: 'Processing',
message: state.message,
showRetry: false, // Do NOT let user retry
};
case 'succeeded':
return {
title: 'Done',
message: 'Payment confirmed.',
showRetry: false,
};
case 'failed':
return {
title: 'Failed',
message: state.reason,
showRetry: true, // Safe to retry
};
case 'reconciling':
return {
title: 'Checking',
message: state.message,
showRetry: false, // Do NOT let user retry
};
}
}
Users can handle uncertainty.
They cannot handle duplicate charges.
The Real Contract
Think of every write endpoint as mapping:
One User Intent
↓
One Durable Outcome
Not:
One HTTP Request
↓
One Outcome
Because requests can be duplicated everywhere.
User Tap
↓
Browser Retry
↓
SDK Retry
↓
Load Balancer Retry
↓
Service Retry
↓
Queue Redelivery
↓
Webhook Replay
Every layer may duplicate.
Your product should not.
// The contract: intent → outcome, not request → outcome
interface IdempotentEndpoint {
// Input: user intent, not raw HTTP
intent: UserIntent;
// Key: stable across retries of same intent
key: string;
// Output: always the same for same key
outcome: Outcome;
}
// Implementation
async function handleIntent(intent: UserIntent, key: string): Promise<Outcome> {
const existing = await lookupByKey(key);
if (existing) return existing.outcome;
const outcome = await executeIntent(intent);
await store(key, outcome);
return outcome;
}
Operation Identity
Every system eventually invents one.
It may be called:
Idempotency Key
Client Token
Request ID
Operation ID
Payment Intent ID
Command ID
Event ID
The name matters less than the lifecycle.
It should be:
Unique enough
Scoped correctly
Persisted
Reused only for retries
Bound to a request fingerprint
Present in logs, traces, support tools, and downstream services
-- Idempotency record lifecycle
CREATE TABLE idempotency_keys (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id UUID NOT NULL,
key TEXT NOT NULL,
request_hash TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'pending', -- pending | succeeded | failed
response JSONB,
resource_id UUID,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
expires_at TIMESTAMPTZ NOT NULL,
UNIQUE (tenant_id, key)
);
-- Auto-cleanup: keys expire after 24 hours
CREATE INDEX idx_idempotency_expires ON idempotency_keys (expires_at)
WHERE status = 'pending';
Support teams love operation IDs.
Instead of asking:
"Can you explain what happened?"
they search one identifier.
Everyone sees the same story.
A Practical Implementation
1. Client Generates the Key
Generate it when the user begins the operation.
Not every retry.
// Client side - key lifecycle
function onCheckoutClick() {
// One key per user action
const key = crypto.randomUUID();
sessionStorage.setItem('checkout-key', key);
api.post('/checkout', cart, {
headers: { 'Idempotency-Key': key },
});
}
function onRetryClick() {
// Reuse existing key
const key = sessionStorage.getItem('checkout-key');
if (!key) return redirectToCheckout();
api.post('/checkout', cart, {
headers: { 'Idempotency-Key': key },
});
}
2. Store an Idempotency Record
-- Server side - record creation
BEGIN;
-- Check for existing key
SELECT status, response FROM idempotency_keys
WHERE tenant_id = $1 AND key = $2;
-- If found: return stored response
-- If not found: insert and process
INSERT INTO idempotency_keys (tenant_id, key, request_hash, status, expires_at)
VALUES ($1, $2, $3, 'processing', now() + interval '24 hours')
ON CONFLICT (tenant_id, key) DO NOTHING;
-- Process the request
INSERT INTO orders (...) VALUES (...) RETURNING id;
-- Update key with result
UPDATE idempotency_keys
SET status = 'succeeded', response = $4, resource_id = $5
WHERE tenant_id = $1 AND key = $2;
COMMIT;
3. Validate Key Reuse
// Server middleware - idempotency check
async function idempotencyGuard(req: Request, key: string) {
const existing = await db.query(
`SELECT status, response, request_hash FROM idempotency_keys
WHERE tenant_id = $1 AND key = $2`,
[req.tenantId, key]
);
if (existing) {
if (existing.status === 'succeeded' || existing.status === 'failed') {
// Finished - return stored response
return new Response(existing.response, { status: existing.status === 'succeeded' ? 200 : 500 });
}
if (existing.status === 'processing') {
// Still running - return 409 or 202 depending on API contract
return new Response(JSON.stringify({ status: 'processing' }), { status: 409 });
}
}
// New request - store and continue
const requestHash = hashPayload(req.body);
if (existing && existing.requestHash !== requestHash) {
return new Response(
JSON.stringify({ error: 'Key reused with different payload' }),
{ status: 422 }
);
}
return null; // Continue to handler
}
4. Make Writes Atomic
Bad:
Save key
↓
Order creation fails
Retry now sees the key.
No order exists.
Also bad:
Create order
↓
Save key fails
Retry creates another order.
Use one database transaction.
// Atomic idempotency + write
async function createOrderWithIdempotency(key: string, order: OrderData) {
return db.transaction(async (tx) => {
// 1. Insert key atomically
const inserted = await tx.query(
`INSERT INTO idempotency_keys (tenant_id, key, request_hash, status, expires_at)
VALUES ($1, $2, $3, 'processing', now() + interval '24 hours')
ON CONFLICT (tenant_id, key) DO NOTHING
RETURNING id`,
[order.tenantId, key, hashPayload(order)]
);
if (!inserted) {
// Key exists - fetch and return stored result
const existing = await tx.query(
`SELECT status, response FROM idempotency_keys
WHERE tenant_id = $1 AND key = $2`,
[order.tenantId, key]
);
return JSON.parse(existing.response);
}
// 2. Create order in same transaction
const result = await tx.query(
`INSERT INTO orders (...) VALUES (...) RETURNING *`,
[order]
);
// 3. Update key with result
await tx.query(
`UPDATE idempotency_keys SET status = 'succeeded', response = $3, resource_id = $4 WHERE id = $1`,
[inserted.id, JSON.stringify(result), result.id]
);
return result;
});
}
5. Handle External Side Effects
Database transactions cannot roll back:
Email
Kafka
Stripe
Slack
Webhooks
Use the Outbox Pattern.
-- Inside the same transaction as the order
INSERT INTO outbox_events (id, aggregate_type, aggregate_id, event_type, payload)
VALUES (
gen_random_uuid(),
'order',
$order_id,
'order.created',
jsonb_build_object('orderId', $order_id, 'email', $customer_email)
);
Worker:
// Outbox worker - idempotent side effects
async function processOutboxEvent(event: OutboxEvent) {
// Check if already processed
const processed = await db.query(
`SELECT id FROM processed_events WHERE event_id = $1`,
[event.id]
);
if (processed) return; // Already handled
switch (event.eventType) {
case 'order.created':
await sendOrderConfirmation(event.payload);
break;
case 'payment.captured':
await notifyFinanceTeam(event.payload);
break;
}
// Mark as processed
await db.query(
`INSERT INTO processed_events (event_id, processed_at) VALUES ($1, now())`,
[event.id]
);
}
If the worker crashes, resume from durable state.
Not from scratch.
Exactly Once Is Usually Marketing
Exactly-once processing rarely exists.
Messages duplicate.
Consumers restart.
Providers timeout.
Networks lie.
What users actually want is:
Effectively once.
Maybe a message is processed three times.
But:
One order exists.
One payment is captured.
One welcome email is sent.
That is what "once" means.
// Effectively-once pattern
async function handleMessage(msg: QueueMessage) {
const idempotent = await checkIdempotency(msg.id);
if (idempotent.alreadyProcessed) {
console.log(`Skipping duplicate message ${msg.id}`);
return;
}
await processMessage(msg);
await markProcessed(msg.id);
}
The distinction matters for engineering decisions.
You stop chasing the impossible (true exactly-once delivery).
You start building what users actually need (effectively-once outcomes).
Where Teams Miss It
Payments get attention.
Many other operations deserve the same treatment.
Account Creation
Retry should return the existing account.
Not:
Email already exists.
// Bad: duplicate accounts
try {
await db.users.create({ email, name });
} catch (err) {
if (err.code === 'DUPLICATE') {
throw new Error('Email already exists'); // User confused
}
}
// Good: idempotent creation
const key = `signup-${email}`;
const result = await idempotentUpsert(key, async () => {
return db.users.create({ email, name });
});
// Returns existing account on retry - no confusion
Infrastructure Provisioning
One click.
One cluster.
Not two.
Notifications
Webhook replay should not send five emails.
Permissions
Retrying an invite should not generate multiple invite links.
Imports
Retrying CSV uploads should not duplicate every customer.
Subscriptions
Upgrade should not apply discounts twice.
AI Agents
This one is becoming critical.
Agents retry.
Tools fail.
Workflows resume.
If agents:
Create tickets
Send emails
Deploy services
Issue refunds
every action needs operation identity.
// Agent action with idempotency
async function agentAction(action: AgentAction) {
const key = `agent-${action.workflowId}-${action.stepId}`;
// Agent retries after tool failure - same key, same result
return idempotentRequest(key, () => executeAction(action));
}
Agent reliability is an idempotency problem in disguise.
What To Measure
Track:
Idempotency key reuse
Duplicate requests prevented
Parameter mismatch rate
Stuck processing operations
Unknown external outcomes
Reconciliation time
Duplicate downstream side effects
Duplicate order support tickets
Retry success rate
-- Idempotency health dashboard
SELECT
date_trunc('day', created_at) AS day,
COUNT(*) AS total_keys,
COUNT(*) FILTER (WHERE status = 'succeeded') AS succeeded,
COUNT(*) FILTER (WHERE status = 'failed') AS failed,
COUNT(*) FILTER (WHERE status = 'processing') AS stuck,
COUNT(*) FILTER (WHERE expires_at < now() AND status = 'processing') AS expired_stuck,
AVG(EXTRACT(EPOCH FROM (updated_at - created_at))) AS avg_resolution_seconds
FROM idempotency_keys
WHERE created_at > now() - interval '7 days'
GROUP BY 1
ORDER BY 1 DESC;
Metrics reveal product fragility.
The Idempotency Checklist
Before shipping a write endpoint, ask:
What is one logical user intent?
Which retries belong to that intent?
Who generates the operation key?
What scope prevents collisions?
How long should keys live?
What happens when the same key has different payload?
What should callers receive while work is in progress?
What happens after timeout with unknown outcome?
Can support locate operations using the key?
Are downstream systems also idempotent?
What does the UI say during uncertainty?
Can users retry safely?
If these answers are unclear,
the endpoint is not finished.
The Boring Feature Nobody Praises
Nobody opens a support ticket saying:
Loved how your API returned the same result after packet loss.
They notice the opposite.
They notice:
Money disappearing twice.
Duplicate emails.
Dashboard saying failed even though it worked.
Support saying:
We don't know what happened.
Idempotency is invisible when it works.
Like:
Backups
Seatbelts
Autosave
Invisible does not mean optional.
It is trust infrastructure.
Reliable products do not merely survive retries.
They make retries feel normal.
That is the standard worth building toward.
Users can click again.
Networks can fail.
Workers can restart.
Messages can replay.
And the product still tells one coherent story.