Making an integration idempotent so retries are safe

Written By: BrainSoft In Backend

You've built an integration that sends orders to a fulfillment service. One day the network hiccups, the request times out, and your code retries. Now the customer gets two shipments. Retries are necessary for reliability, but without idempotency they can cause chaos. The fix is to make every operation safe to repeat.

Idempotency means that performing the same operation multiple times has the same effect as doing it once. To make your integration idempotent, you attach a unique key to each request, check if that key was already processed, and ensure your side effects are atomic. This post walks through the steps to implement that reliably.

Why retries break things

When you send a request and don't get a response, you don't know if the server processed it. The request might have failed before reaching the server, or the server might have completed the work but the response got lost. If you retry, you risk duplicating the action. This is especially dangerous for operations that create resources, charge payments, or send notifications.

The solution is to give each logical operation a unique identifier—an idempotency key—and have the server use that key to detect and ignore duplicates. The client generates the key once and reuses it for all retries of the same operation.

Step-by-step: making an integration idempotent

1. Generate a unique idempotency key. Use a UUID or a deterministic hash of the request payload. The key must be unique per operation but stable across retries.

   import uuid
   idempotency_key = str(uuid.uuid4())

2. Store the key before making the request. Save the key and the request state in your database. This ensures you can check later if the operation was already attempted.

   INSERT INTO idempotency_keys (key, status) VALUES ('...', 'pending');

3. Send the key with the request. Include it in a header like Idempotency-Key or in the request body. The server must use it to deduplicate.

   headers = {'Idempotency-Key': idempotency_key}
   response = requests.post(url, json=payload, headers=headers)

4. Handle the response and update the key status. If the request succeeds, mark the key as completed. If it fails, you can retry with the same key.

   if response.ok:
       UPDATE idempotency_keys SET status = 'completed' WHERE key = '...';

5. On retry, check the key first. Before sending a new request, look up the key. If it's already completed, skip the operation and return the stored result. If it's pending, you might need to wait or check the server's status endpoint.

   SELECT status, result FROM idempotency_keys WHERE key = '...';

6. Make the server side idempotent. The server must enforce uniqueness. Use a database constraint or a dedicated idempotency table. When a request with a known key arrives, return the original response without re-executing.

   CREATE TABLE idempotency (key TEXT PRIMARY KEY, response JSONB);

7. Set an expiration for keys. You don't need to keep keys forever. A TTL of 24 hours is common. After that, the same key could be reused, but that's unlikely in practice.

   DELETE FROM idempotency_keys WHERE created_at < NOW() - INTERVAL '24 hours';

8. Test with duplicate requests. Write tests that send the same request twice and verify that only one action occurs. Use tools like curl or your test suite to simulate retries.

   curl -X POST -H "Idempotency-Key: abc123" ...

Common pitfalls and how to avoid them

Even with idempotency keys, things can go wrong. Here are a few traps:

• Not storing the key before the request. If you generate the key, send the request, and then store it, a crash between send and store means you can't retry safely. Always persist the key first.
• Using the same key for different operations. Each logical operation needs its own key. Don't reuse keys across different payloads.
• Assuming the server supports idempotency. Not all APIs do. If the server doesn't support idempotency keys, you'll need to implement a deduplication layer on your side or use a different approach.
• Forgetting about side effects. If your integration triggers emails, webhooks, or other side effects, those need to be idempotent too. For example, use a message queue with deduplication.

If you're building integrations that must be reliable, consider working with a team that has done it before. At BrainSoft, we help clients design and implement idempotent integrations as part of our services.

Frequently asked questions

#Backend