Connecting two systems without losing data on the way
Retries, idempotency and traceability: the three things that separate an integration that works from one that looks like it works.
Almost every broken integration I have had to fix started the same way: someone wrote a script that reads from one side and writes to the other. It worked. They left it running. Six months later there are invoices missing and nobody knows since when.
The script was not badly written. It was missing three things nobody asks for in the kickoff meeting, because they are invisible until they fail.
1. The network fails, and it fails at the worst moment
When your code calls an API there are three possible endings: it worked, it failed, or you do not know. The third one is the interesting one.
If the connection drops after the request went out but before the response arrives, your side logs an error and the other side logs a new record. You retry, and now there are two invoices.
The answer is not “retry less”. It is retrying with your head on:
- Exponential backoff with jitter. If you retry every second, when the other system goes down all your processes hammer it together and stop it coming back up. Wait 1 s, 2 s, 4 s, 8 s, and add up to a second of randomness so they do not synchronise.
- A ceiling. After N attempts the message goes to a separate queue and someone looks at it. Retrying forever is hiding the problem.
- Know what to retry. A 503 gets retried. A 400 does not: the request is wrong and it always will be. Retrying a 400 is spending resources to fail later.
2. Idempotency: being able to repeat without breaking
If the previous point leaves you retrying, you need repeating to be harmless. That is called idempotency and it is the piece most often skipped.
The practical form: every operation carries its own stable key, derived from the data and not from the moment. The invoice number works. A UUID generated before sending works too. Date.now() does not, because it changes on every attempt.
The receiver stores those keys and, if it sees a repeat, returns the previous result instead of creating something new. If the receiver is somebody else’s system that does none of this, you do it: before creating, you ask whether it already exists. It is one more call and it saves you the duplicate.
An idempotent integration can be re-run end to end after a disaster. One that is not cannot be touched without fear, and fear is why nobody fixes it.
3. Traceability: being able to answer “what happened to order 4412?”
This is the one that will save your relationship with the client.
Every message that crosses carries a correlation id that travels end to end and appears in every log line. When someone asks about an order, you search for that id and see the whole story: when it came in, how many attempts there were, what the other side returned, when it was accepted.
Without that, the honest answer to “what happened to 4412?” is “I don’t know”, and that answer costs more than the whole build.
Two details that matter: store the request and response bodies, at least for a while, because “it returned an error” is useless without the message; and do not store personal data you do not need, because logs are also a place where things leak.
The minimum that lets you sleep
If you are about to connect two systems and you are short on time, this is the order:
- An idempotency key on every operation.
- Retries with backoff and a dead-letter queue.
- A correlation id in every log line.
- An alert for when the dead-letter queue stops being empty.
Four things, done in days rather than months, and they are the difference between an integration that lets you sleep and one that has you checking spreadsheets on Monday mornings.
The rest — dashboards, metrics, automatic reprocessing — comes later, and only gets built if these four are there.