In Part 1, Ivan (Principal Engineer with the Deposit & Savings team) walked through the three approaches that quietly power most scheduled jobs in production. This second part picks up at the point where every one of those simple solutions eventually fails.

Haven't read Part 1 yet? Start here

When simple solutions are no longer enough

And then our product grew; deposits became far more numerous, which is great news. The bad news is that our single-instance processor could no longer cope. As always, it happened unexpectedly and under the banner of "well, who could have predicted that?"

Yet, the symptoms are familiar:

• Execution time grows linearly with the data volume and stops fitting into the interval between ticks — the next run starts while the previous one hasn't finished
• A heavy job blocks the node, and lightweight jobs sit idle
• A crash mid-processing means redoing the work from the very beginning
• Where exactly it got stuck is hard to tell from the logs

At this moment, it becomes visible that the job is actually two fundamentally different tasks packed into one method:

• Find the records that need to be processed
• Process them

It makes sense to scale the second part. The first stays lightweight: even at large volumes, "find 100 thousand ids" is cheaper than "process each one." This separation is what opens up the space for everything discussed below.

Reader + Processor: scaling the processing

Reader → Kafka → consumers

The scheme: one instance selects work from a source (a database, an external API, anything) and publishes tasks to Kafka. Several consumers process the messages in parallel.

Pros:

• Natural backpressure through consumer lag
• Consumers scale independently of the reader
• Retry and DLQ are standard practice — it's all been figured out
• Full separation of reading and processing; you can update and deploy them independently

Cons:

• Kafka is an infrastructure you have to have and maintain
• Ordering only within a partition — if order by key matters, design the partitioning up front
• Exactly-once delivery is unachievable in the general case — a message may arrive more than once (producer retry, consumer rebalance, redelivery after a failure). Therefore, the handler must be idempotent: reprocessing the same message must not change the result. That's what gives you exactly-once at the effect level — which is what people usually mean when they say "exactly-once." Kafka transactions cover the atomicity of "read → process → commit offset," but they don't relieve you of business-operation idempotency
• Lag monitoring becomes a first-class concern

Reader → REST

The scheme: one instance selects work and calls the handler's REST endpoint in parallel — either a single one or several directly.

Pros:

• No Kafka — operationally simpler
• Synchronous feedback: the reader immediately knows whether it was processed or not
• Direct control over parallelism through the call-pool size

Cons:

• The reader implements retry, timeout, and backpressure itself
• A slow handler blocks a connection in the reader
• Not suitable for large volumes — at scale, it turns into a homemade Kafka
• No natural replay/audit

The shared problem of both approaches: the reader as a SPOF

In both the Kafka variant and the REST variant, the reader is a single instance with a latch (the same as ShedLock or leader election). If it crashes mid-batch-read, unpleasant questions arise:

• Which records have already been sent for processing and which haven't?
• Will they be guaranteed to be resent on the next tick?
• Won't we send them twice?

These questions are solvable (idempotent reader, checkpoint after publishing, transactional outbox), but they haven't gone anywhere — they're simply hidden in the architecture. Next, we'll look at how to get rid of the reader-as-SPOF entirely.

Getting rid of the reader as a SPOF

DB-level partitioning

The main idea is to remove the reader as an explicit place in the architecture. Each worker pulls their own share of records directly from the database. There's no coordinator: the database serves as the arbiter.

There are two approaches, each with its own set of limitations.

SELECT ... FOR UPDATE SKIP LOCKED

In PostgreSQL (since 9.5) and MySQL (since 8.0), you can atomically "grab" free records: the query locks the selected rows and skips the ones already locked by another transaction. Several workers run in parallel:

SELECT * FROM jobs

WHERE status = 'pending'

LIMIT N

FOR UPDATE SKIP LOCKED

And each one gets its own non-overlapping set. Coordination is free, at the DBMS level. This is elegant, but there are two important "buts":

1. Dependence on the DBMS. FOR UPDATE SKIP LOCKED is supported in identical form by PostgreSQL (since 9.5), MySQL (since 8.0), and Oracle (since 11g — Oracle Advanced Queuing is built on exactly this). SQL Server assembles the same behaviour via the WITH (UPDLOCK, READPAST) hints; Google Spanner has no such mechanism at all; CockroachDB gained it relatively recently but with a number of caveats (column families, retries under SERIALIZABLE, contention). Before relying on it, check the current limitations for your version.

2. Processing must happen within a transaction. The lock is released on commit or rollback — while it's held, no one else will touch those records. If the processing is short and local, all is well. In our deposit example, this works while crediting interest is just an UPDATE to the balance. But as soon as, before crediting, you need to go to an AML service, fire a notification, or update the balance in a downstream system, keeping the transaction open during a network call is a bad idea: long transactions hit database performance, it's easy to catch a timeout, and it's easy to hang for a long time. This is usually cured by a two-phase pattern — a short transaction marks the records "in progress with worker X" and commits, then the processing happens outside the transaction, then a final transaction marks them "done." But that's no longer SKIP LOCKED in its pure form; it's a homemade queue on top of a table where SKIP LOCKED is used only at the grab stage.

Hash-based partitioning + Quartz

An approach that doesn't depend on the DB dialect and doesn't run into transaction length. The idea is to distribute records across shards in advance and bring up a separate reader for each shard.

A shard_id column is added to the table, computed as hash({partition_key}) % N and saved when the record is inserted. In our deposit example, partition_key is, say, deposit_id — each deposit independently lands in its own shard. That is, sharding happens at write time, not at read time — an index on shard_id makes reading almost free.

The architecture then looks like this:

• N triggers are configured in Quartz — one per reading service. A Quartz cluster is convenient in this role: it can distribute triggers across nodes and provide their failover
• N reader deployments are rolled out in k8s, each with its own shard_id in the configuration
• Each reader hooks onto its own trigger and independently reads only its own records (WHERE shard_id = X)
• From there, each reader works exactly as in the "reader → Kafka → consumers" scheme (see the "Reader + Processor" section): it publishes tasks to Kafka, and the consumers handle them.

In practice, the number of shards is often taken to be larger than the number of readers — for example, 64 shards and 4 readers, each responsible for 16 shards. This gives headroom for scaling: adding new readers doesn't require recomputing shard_id.

The main virtue of this approach is the migration path. If a project already runs the classic Quartz + Kafka + handler scheme (that is, everything discussed in the "Simple solutions" and "Reader + Processor" sections), the transition is made without re-architecting: a shard_id column is added, the reader configuration is duplicated into N instances with different values, and N triggers are configured. The handlers (Kafka consumers) don't change at all. Evolution, not revolution.

Among the limitations:

• The number of shards is chosen at the outset. Changing it later means recomputing shard_id across all existing records. That's why shards are usually taken with headroom
• When inserting a new record, you need to compute shard_id correctly. This is either computed in application code, or in a DB trigger, or the column is made generated/computed (if the DBMS supports it)
• N reader services mean N deployments, each with its own alerting and dashboard. In practice, this isn't scary (labels help), but the operational footprint is larger than with a single reader. This is partially mitigated by a k8s StatefulSet: instead of N separate deployments, a single object with N pods that have stable ordinal indexes (reader-0, reader-1, …), and shard_id maps directly onto that index. It's important to understand that a StatefulSet reduces the operational footprint but does not solve the problem below: if a pod crashes, k8s will restart it with the same index, and for the duration of the downtime the shard still sits idle
• There's no rebalancing of shards across handlers. This is the main risk of the approach, and it's easy to miss behind the words "the Quartz cluster provides failover." Quartz will indeed pick up the trigger of a fallen node — but the shard → reader assignment is static, it's nailed to the pod's config. So if the handler of shard X itself goes down (a bad deploy, OOM, a crash loop), its trigger is alive and ticking, but there's no one to read WHERE shard_id = X: the other readers are bound to their own ranges and won't take someone else's. Shard X sits idle until the pod recovers. The Quartz cluster protects triggers, but not handlers — and here it's precisely the handler that fails

What both approaches have in common

The reader as a SPOF either disappears or is broken up into N independent parts. In SKIP LOCKED — through workers grabbing records directly, without a dedicated reader. In hash + Quartz — through the readers becoming N, where the crash of one paralyses only its shard (1/N of the total work), and the Quartz cluster guarantees that another node picks up that shard's trigger.

If an idle shard is an unacceptable risk, rebalancing has to be added separately — by hand or with a ready-made tool. The DIY path moves shard assignment into a coordinator backed by ZooKeeper or etcd: each reader registers an ephemeral record on startup ("I'm alive, I hold these shards"), and when a pod crashes, its record expires, a watch fires on the others, and the fallen shards are reassigned to the living. Shard → reader becomes dynamic rather than configurational. Apache Curator gives the primitives (LeaderLatch / LeaderSelector for electing a coordinator), but the shard-distribution logic on top you write yourself.

The ready-made path is Apache ShardingSphere ElasticJob (Lite) — the same combination under the hood (Quartz for triggers + ZooKeeper for coordination), but with sharding and rebalancing already built in: slices are redistributed automatically when a node is added or removed, and an optional failover finishes off an interrupted slice within the current cycle. The price is a hard ZooKeeper dependency and lock-in into the framework's Job API, which loses the seamless "Quartz + Kafka" migration this whole approach was built for. Worth a sober look at the project's state, too: the core is barely developed (last functional release end of 2023), and ZooKeeper coordination can itself become a point of failure under network instability. As proof, the problem is solvable out of the box; it's useful; as a default for a new project, debatable.

Spring Batch remote partitioning

Spring Batch is a framework purpose-built for batch processing: reading in batches, chunk-oriented transactionality, retry/skip policies, persistent job state in a JobRepository, restart capability "from the point where you fell." For our task, one specific capability is interesting — remote partitioning.

The idea is simple but important: the manager doesn't read the data. The manager only describes how to split it. Concretely, it works like this:

• The job has a Partitioner — a component that produces N "execution contexts." Each context is a dictionary of parameters, for example, {minId: 0, maxId: 9999}, {minId: 10000, maxId: 19999}, and so on.
• The manager dispatches these contexts to the workers through a broker (Kafka, RabbitMQ, JMS — Spring Integration can work with any of them)
• The workers listen to the queue. Each one takes a single context and launches at home a step whose ItemReader is parameterised by these minId/maxId — that is, it pulls only its own range from the database
• The worker processes its partition fully (read + process + write) and sends a response to the manager

• The manager waits for responses from all partitions and closes the job

The fundamental difference from "reader → Kafka" (see the "Reader + Processor" section): the data doesn't fly over the network. Only the metadata "go read this range" flies over the network. The records themselves are pulled by each worker directly from the database.

What this gives you:

• The reader as a SPOF disappears: the manager does only cheap work — slicing the ranges. The reading of data is distributed
• Restart at the partition level: 2 of 10 workers failed — on retry, only those 2 restart, not the whole job
• Declarative scaling: change gridSize from 4 to 16 — get more partitions without rewriting code
• The state of each partition is tracked persistently in the JobRepository: you can see what failed, where, and on what, rather than "it started — it finished" from the @Scheduled world

What hurts:

• The learning curve is steep. Job, Step, JobLauncher, JobRepository, Partitioner, StepExecutionAggregator, PartitionHandler — there are many concepts, and without some time to get up to speed, you can't work productively with the framework
• It's own tables in the database. The story is the same as with Quartz — Spring Batch is also stateful, with a dozen service tables
• A skewed partitioning function is the main pain in practice. If you split by ID ranges, but the actual records are dense in one range and sparse in another, one worker hauls 80% of the work while the rest sit idle. Good partitioning is a separate skill: range, hash, time-based, custom — and it's tuned to the specific data profile
• The manager-worker round-trip adds latency. For short jobs (seconds), the overhead can exceed the benefit. Spring Batch makes sense where it's about minutes and hours of work
• Not for streams. This is about "run through a finite set and process it," not about continuous processing of incoming events

When it's justified: one-off or periodic runs over a large finite set, where restart capability, an audit trail of "what got processed," and partitioning at the level of the task itself — rather than the infrastructure beneath it — matter.

Durable execution: Temporal and its neighbours

Everything discussed above lives inside the JVM world and k8s primitives. But there's a separate class of tools worth keeping in mind — external orchestrators with durable execution: Temporal (and its ancestor Cadence), and partly Airflow and Argo Workflows for batch scenarios.

Why is this relevant to our example specifically? As soon as crediting interest stops being a single UPDATE balance and turns into a chain of "check in AML → credit → update the balance downstream → send a notification," where each step is a network call that may crash, hang, or return more than once, we start reproducing by hand exactly what Temporal gives out of the box:

• Durable execution state. The workflow remembers which step it stopped at. It crashed on the notification after a successful crediting — on retry. Temporal won't credit again, it'll continue from the notification. This removes a significant part of the "don't do it twice" pain, which in DB-centric schemes you have to close with idempotency keys and an outbox by hand
• Retry as a declaration, not as code. The retry policy (backoff, limits, not retrying certain errors) is set by the step's configuration, rather than being written into every handler
• Visibility. The state of each workflow instance is tracked and viewable — this closes that very "where exactly it got stuck" pain we started with

What we pay: a separate piece of infrastructure (a Temporal cluster or Temporal Cloud), its own programming model (deterministic workflow code, separated from the activities), and a noticeable learning curve. For a simple periodic job this is a cannon on sparrows — no less so than Spring Batch. In many teams, adding yet another heavy infrastructure component is more expensive than squeezing out a solution on what already exists (a database, Kafka, Quartz) — which is why the rest of this article stays in the JVM world. But if the core of the task is reliably running multi-step processes with external calls and a strict "exactly once by effect," it's worth looking toward durable execution earlier than erecting a third layer of homemade orchestration on top of a table.

Conclusion

In system design, there's no silver bullet. The same approach will turn out elegant in one project and over-engineered in another.

@Scheduled + ShedLock, a Quartz cluster, or a K8s CronJob cover a huge share of real-world tasks — and dragging Spring Batch in there would be shooting sparrows with a cannon. Where the infrastructure is controllable, you can lean on the capabilities of a specific database: SKIP LOCKED in Postgres or MySQL gives almost-free coordination of workers. And where you need to work with a distributed database or with heavy processing involving external calls, DB-agnostic solutions are needed: hash partitioning with Quartz, Spring Batch with remote partitioning, something else.

Separately, it's worth paying back the debt on observability.

In this article, I've covered only some of the possible options and tried, for each, to indicate where it works well and where it breaks down. The main skill in this area isn't "knowing the single right approach," but matching the characteristics of the task to the characteristics of the solutions and choosing the least bad one for the specific context.

We're looking for engineers like Ivan, who take their time to understand a system deeply enough to know exactly when to stop reaching out for more architecture because the problem in front of them doesn't need more architecture. If you're that engineer, we'd love to speak to you.

Explore open engineering roles at Moniepoint.