Batch Data Pipelines That Resume, Retry and Refresh

Pipelines fail, so design for it

Any job that calls external APIs, downloads files or processes thousands of records will eventually fail partway through: a network drop, a rate limit, a bad document or a server restart. The difference between a frustrating pipeline and a dependable one is how it behaves after failure. Across our data tools we use the same set of habits.

Make each step idempotent

An idempotent step can be run twice with the same result. If processing a company writes its result with an update that overwrites the same row, rerunning it is harmless. Avoid designs where repeating a step creates duplicates or double-counts. Idempotency is what makes retries safe.

Record state in the database

Store the status of every item, such as pending, running, done and failed, together with timestamps and error messages. State in the database survives restarts, can be queried and drives dashboards. A resume command then becomes trivial: process everything that is not yet done.

Retry, but with limits

• Retry temporary failures, such as timeouts and rate limits, with a delay between attempts.
• Do not retry permanent failures forever. After a few attempts, mark the item failed with a reason.
• Provide a deliberate retry-failed action for operators to use after fixing the cause.
• Reinforce the approach on a retry when it makes sense, for example by using a stricter prompt for an AI model.

Refresh stale data on a schedule

Data decays. People change jobs, companies file new accounts and prices move. We schedule refreshes, for example re-checking records older than 30 days or running a monthly job at a quiet hour, and we prioritise records that are missing data. Scheduling is done with the operating system's cron or a service manager, which is simple and dependable.

Protect shared resources

When several processes start at once, they can collide. A database advisory lock around schema creation prevents duplicated setup work. Row locking lets many workers share a queue without processing the same item twice. Background tasks let an API return immediately while a long job continues.

Make progress visible

Provide counts, rates, recent errors and the state of each worker through a simple dashboard or status endpoint. People trust a pipeline when they can see it working, and problems are found sooner.

A checklist

• Can I stop it at any moment and continue later?
• Is every step safe to run twice?
• Do failures leave a clear reason I can read?
• Does old data refresh automatically?
• Can I see what is happening right now?