AI Agents
Retries, crashes and resumed runs make agents repeat side effects like duplicate emails and double charges. Make tools idempotent with keys, a ledger and upserts.
Here is a failure that every team running agents in production meets eventually. The agent calls a tool to send a customer an email. The email provider is slow, the request times out after 30 seconds, and the retry logic tries again. Both requests succeed. The customer gets the same email twice.
With email, that is embarrassing. With a refund, a payment or a published post, it is a real problem. The fix is an old idea from distributed systems: make the action idempotent, so that doing it twice has the same effect as doing it once.
Any system that retries can duplicate side effects. Agents have more ways to retry than most:
The last one is unique to agents. Even with perfect infrastructure, the model can choose to repeat an action. You cannot fix that with retry settings. You fix it in the tool.
Reading is already idempotent. Searching for a ticket twice changes nothing. Some writes are naturally idempotent too: setting a field to a value, adding a label that is already present, overwriting a file with the same content.
The tools that need attention are the ones that create something or trigger something each time they run:
An idempotency key is a unique identifier for an intended action. Before executing, the tool checks whether that key has already been used. If it has, it returns the stored result instead of acting again.
The question is what to build the key from. There are two options, and the difference matters.
Every tool call from the Claude API has an id. It is stable for that call, so if your code retries executing the same tool call, the id stays the same. That catches transport retries and crash-and-resume.
But if the model decides to call the tool again in a later turn, the new call gets a new id. The key changes and the duplicate goes through.
A more robust key is built from the run and the meaning of the action: which run, which tool, and the arguments that define what the action does.
import { createHash } from 'crypto';
function actionKey(
runId: string,
tool: string,
input: Record<string, unknown>,
) {
// Only the fields that define the action. Leave out free text the
// model may rephrase between attempts, such as a note or a summary.
const identity = IDENTITY_FIELDS[tool].map((f) => [f, input[f]]);
return createHash('sha256')
.update(JSON.stringify([runId, tool, identity]))
.digest('hex');
}
const IDENTITY_FIELDS: Record<string, string[]> = {
send_reply: ['ticketId'], // one reply per ticket per run
refund_order: ['orderId', 'amount'],
publish_post: ['draftId', 'channel'],
};
Choosing the identity fields is a product decision. For a support agent, "one reply per ticket per run" might be exactly right. For a newsletter agent, a reply to the same person on a different topic should be allowed, so the topic belongs in the key. Think about what "the same action" means for each tool, and write it down in code.
The key is only useful if you check it. Keep a ledger of completed actions in durable storage, the same database that holds your run state.
async function executeOnce(runId: string, call: ToolCall) {
const key = actionKey(runId, call.name, call.input);
// Claim the key atomically. If the insert fails, someone else has it.
const claimed = await ledger.tryInsert({ key, status: 'in_progress' });
if (!claimed) {
const existing = await ledger.get(key);
if (existing.status === 'done') {
const note = 'Already done earlier in this run: ' + existing.summary;
return result(call.id, note);
}
return result(call.id, 'Already in progress. Do not retry.', true);
}
try {
const run = TOOLS[call.name];
const outcome = await run(call.input, { idempotencyKey: key });
await ledger.update(key, { status: 'done', summary: outcome.summary });
return result(call.id, outcome.summary);
} catch (err) {
await ledger.update(key, { status: 'failed', error: String(err) });
throw err;
}
}
The claim has to be atomic: a unique constraint on the key column, a conditional write, or a transaction. A check followed by a separate insert leaves a window where two parallel attempts both see "not used" and both proceed.
Notice the messages returned when a key has already been used. "Already done earlier in this run" is not an error. It is information. Without it, a model that retried because of an unclear result might try a third time, or conclude something is broken and report a failure to the user.
Clear tool results are part of idempotency. The model should be able to tell the difference between "this failed", "this succeeded" and "this was already done". Our article on tool result error handling goes deeper on writing results the model can act on.
The ledger protects against your own system repeating an action. It cannot help if the request reaches the external service, succeeds, and the response is lost before you record it. For that, the external service itself has to recognise the repeat.
Many payment and messaging APIs accept an idempotency key with each request. Stripe, for example, takes an Idempotency-Key header and returns the original response for a repeated key instead of creating a second charge. Where a service supports this, always pass your action key through.
Where it does not, use a reference you control:
Idempotency bugs only appear when things go wrong, so test the wrong cases deliberately:
These belong in your regular test suite. Our article on agent testing strategies covers how to fake external services so you can trigger these cases reliably.
An idempotency layer is a few dozen lines of code and one database table. In return, every retry, crash and resume in your agent becomes safe by default. It also makes other features easier: you can retry aggressively on transient errors, and resume runs after an approval without worrying about double execution. Pair it with error recovery and you have an agent that can fail halfway through a task and pick up where it left off without anyone noticing.