Scaling Data Extraction with a Postgres Queue and Parallel Workers

When one process is not enough

Extracting data from thousands of filings takes time, and each call is limited by external rate limits. A single script that loops over companies is slow and fragile: if it stops halfway you have to work out where it was. We needed a way to run many extractions in parallel, spread across several API keys, and to recover cleanly from failures.

The design: a queue in Postgres

Rather than adding a separate message broker, we used a table in Postgres as the work queue. Each row is a company to process, with a status such as pending, in progress, done or failed. Many small workers pull rows from the same table, each holding its own credentials, call the extraction service and write results back to the row. A small admin API serves a live dashboard, worker status and a CSV export.

Why Postgres is a good queue for this

• It is already part of the stack, so there is nothing new to operate.
• Workers can claim rows safely with row locking, so two workers never process the same company.
• Results and status live together, which makes reporting and exports simple.
• Retrying failed items is a single update.
• You can inspect the queue with ordinary SQL.

Workers that are easy to reason about

Each worker is deliberately small. It claims a batch of rows, processes them one by one, records the outcome and repeats. Because every worker has its own set of keys, rate limits are shared out rather than multiplied against a single account. Adding capacity means starting another worker. Removing one is safe, because unfinished rows are simply picked up again.

Visibility is a feature

A long-running job is only manageable if you can see it. The dashboard shows how many rows are pending, running, finished and failed, which workers are alive and how fast they are going. A retry-failed action lets an operator re-queue problems after fixing the cause. We also added a monitor that watches for risks such as stalled workers.

Running it in production

The API and workers run as managed services that restart automatically, with configuration kept in environment files rather than in code. Deployment notes and migrations are kept alongside the code, so a new server can be set up repeatably.

When this approach fits

A database-backed queue is an excellent choice for moderate volumes where reliability and simplicity matter more than extreme throughput. If you later need much higher scale, the same pattern can move to a dedicated queue system with little change to the workers. Planning data pipelines or background processing for your business? We can help design it.