Generating a batch of PDFs requires more than looping over a render function. The process can stop after the service accepts a job but before you save its ID. A webhook can arrive twice. A temporary download can expire before an operator resumes the batch.
The downloadable Python example uses a SQLite queue, two worker threads, stable idempotency keys, and atomic output writes. It includes a separate signed-webhook inbox. The two-document fixture was run against staging and then run again to confirm that saved documents were not submitted again. Download the example projects, fixtures, and results.
Start with a durable manifest
Each manifest entry has a stable business ID and a path to an HTML file:
[
{"id": "invoice-usd", "html": "invoice-usd.html"},
{"id": "invoice-eur", "html": "invoice-eur.html"}
]From the archive root, after Python installation and CLI authentication:
python batch.py fixtures/batch.json work/batchThe queue stores the exact request body and its state in work/batch/queue.sqlite. It rejects an attempt to reuse a business ID with different HTML. Give a revised document a new versioned ID so an operator can tell an intentional regeneration from a retry.
The example holds an operating-system lock while a runner owns the queue. It is designed for one machine and requires POSIX file locking. A distributed deployment needs database claims or leases; do not share this SQLite file between independently running containers.
Submit once, then track the accepted job
api.py submits to /v1/pdf with response: async. The idempotency key is derived from the route and complete request body. If a response is lost after acceptance, retrying the same payload and key allows the server to identify the prior submission.
Persist the returned job ID before moving on. The sample stores it transactionally, then polls /v1/jobs/:id. Its state transition is pending, processing, and saved. A saved row means the PDF has been downloaded and atomically renamed into place, not merely that the remote render finished.
Review the idempotency reference for supported response modes and retention semantics. Idempotency is bounded server state, not an unlimited archive. If you resume after that state expires and never saved the original job ID, reconcile the outcome before generating another document.
Bound concurrency and submission rate separately
Two workers can overlap waiting and downloading, but submissions are spaced at least three seconds apart. Concurrency limits the amount of work in progress; pacing limits how quickly requests arrive. One control does not replace the other.
The client respects Retry-After on retryable responses, including numeric seconds and HTTP-date values. Otherwise it uses bounded exponential delay with jitter. It retries a POST only when it has a stable idempotency key, and stops after six attempts.
Do not retry invalid input unchanged. Authentication failures, insufficient credit, and idempotency conflicts need specific handling. If polling times out, retain the job ID and resume later. A local timeout is not proof that a remote job failed.
Save outputs before marking work complete
The temporary download URL is not your document archive. The worker downloads the bytes, verifies a PDF signature, writes a temporary file, and renames it into place. Only then does it mark the queue row saved.
The local filesystem is the durable storage target in this tutorial. In a deployed application, use durable object storage and store the object's key with the queue record. Keep the same ordering: successfully store the artifact before recording completion.
If a download expires, the example fails visibly and retains the job ID. It does not silently submit a new billable job. An operator can decide whether to recover an existing artifact or intentionally regenerate with a new version.
Add webhooks as durable notifications
webhook.py exposes /webhooks/relaypdf on loopback port 8090. Deploy it behind an HTTPS reverse proxy before registering an account webhook. Set the signing secret returned when creating that webhook as RELAYPDF_WEBHOOK_SECRET.
python webhook.py work/events.sqliteThe handler verifies the HMAC over the original request bytes, checks the timestamp window, and commits the payload to SQLite before acknowledging with 204. Repeated identical payloads share a digest and are stored once. Job-level state transitions must still be idempotent because logically repeated events need not have byte-identical payloads.
This inbox does not replace the queue's polling loop. A consumer can use it to wake a waiting worker sooner, then read authoritative job state. Keeping polling available handles missed deliveries and delayed notifications. The example's signature tests reject tampered and expired payloads.
Know what the example proves
The recorded staging run saved both fixture PDFs. A second invocation performed no new submissions. Automated tests also check that changed HTML under an existing ID is rejected and that altered webhook bytes fail verification.
This is not a throughput benchmark or a complete distributed queue framework. Before scaling it, add claim leases, operator-visible failure states, storage monitoring, and reconciliation for expired artifacts. For the request-level concepts, read async jobs and webhooks. The HTML-to-PDF product page and async reference describe the generating API used here.