As a developer, launching a full ecosystem is crucial to test changes properly. However, as dependencies grow, these systems become hard to launch locally. Either because they consume a lot of resources or because they're complex to launch. This pushes developers to either not test properly or rely on shared environments (like dev), which can impact their team members.\nAt Moniepoint, we built a tool that automates resource deployment on a remote cluster, helping developers test features efficiently with minimal configuration.\nIt’s as straightforward as running an eshu workload with a clear, YAML-based workload definition, making the process accessible and manageable.\n\nLaunching a workload\nAfter users install our custom CLI, the first time they run \neshu workload run\n, it generates a random workload name/ID for the run. This ensures users don't clash with other users' workloads on the same cluster. This ID is stored in a local config file, so that we can refer back to it when needed. Next, the system loads local YAML files to create the workload. Users write these files based on the \nScore\n specification. Each YAML file contains 5 parts: apiVersion, metadata, containers, service and resources. As an example, let’s examine this workload file below:\n\n
The apiVersion identifies the version of the workload specification. The metadata portion lets you add annotations and similar metadata to the created resources, and it also specifies the workload name so other workloads can reference it. The containers section includes the images and environment configuration for our app, while the services section specifies which ports we want to expose. The resources part includes the standard resources we want to launch with the workload. Examples include Postgres or MySQL databases, Redis, Kafka, and many others.\nLet’s take a look at another example. The goal is to create a chat app that includes a front-end component, a backend API, and a backend identity service for user identities. The chat app includes a dedicated database and Redis for caching, while the identity service has its own database.\nTo do this, we will launch the workload file above for the identity service, plus another that includes 2 containers for the front-end and backend components, plus a Redis and Postgres resource. The chat file also includes a resource that references the identity workload.\nWhen the eshu workload runs, it merges the files into a single file. Then file references are resolved. Although the example inlines the database script, you can also import it from a separate file with a file reference (${file:initscript.sql}). After this injection, the CLI validates the workload contents against a predefined schema. This helps the user quickly correct mistakes without making any request to the cluster.\nAfter we have a valid merged YAML definition, the CLI sends the workload through an HTTP request to our server in the remote Kubernetes cluster. This lets us create it with a single workload run command.\nWhen the server receives the request, it first creates a namespace named after the workload ID using the Kubernetes API. It then copies any relevant secrets, like the secret used to pull images from the internal Moniepoint Artifactory, into this namespace. \nIn our example, \nfe65c057-45760157-careful-weasel \nis the name of a dedicated namespace where our resources exist.\n\nAfter creating the namespace, Crossplane creates the XWorkload custom resource from our YAML definition in the Score Workloads attribute in the metadata section.\nCreating this resource kicks off the Crossplane composition pipeline we configured.\n\nComposition Pipeline\nThe XWorkload includes the YAML file contents in the Score Workloads section. With it, Crossplane calls our custom function for each workload created, and our function parses that score file and outputs the relevant Kubernetes resources.\nUsing the \ncrossplane function-sdk-go\n, the function needs to implement the RunFunction signature, receive the current request with context, and return the desired resources response.\nOur custom function has 4 main tasks:\nIt reads and parses the YAML file contents from the request (present in the score workloads attribute). \nAfterwards, it adds the required Kubernetes resources (deployments, volumes, …) to the response. It generates a map of environment variables for all resources so it can inject them into the containers. This map includes the database username and passwords, which the function generates automatically. \nThen the containers are added. \nFinally, the required services are created to expose the resources and the proxy so it is reachable from their machine.\nThe main effort with this function was supporting more and more resources over time. Beyond the resource itself, we also added auxiliary components to ensure users had the same functionality across different resources. For example, while PostgreSQL has an easy way to initialise data via setting up a script file when mounting, the Google Spanner emulator does not. So a job is launched after the emulator starts to connect to the pod and run the provided initialisation script. For the user, there is no difference in how to initialise one or the other. To manage this complexity, each resource implements the same interface:\n\n
For our Postgres example above, the implementation will create a secret with randomly generated credentials and store that secret in Vault, create a config map for the SQL initialisation script, create a service to expose the Postgres pod, and create a stateful set that includes a PVC for persisting data. Since this is Postgres, the config map is mounted into the container at\n/docker-entrypoint-initdb.d/init.sql\n, and the generated credentials are passed as env vars to initialise the user. For other resources, like Spanner, a job initialises the data so the user has a unified interface. \nThen the actual function flow is a simple chain of responsibility to generate all the resources:\n\n
With these 2 simple abstractions, we can extend the function to support more and more resources with minimal changes.\nRegarding the environment, each resource adds to the env map. Then, these values are injected into the containers via string interpolation, replacing the string \nresources.[name].[key]. \nFor example, in our demo above, we had a Postgres database resource called pg. The container's environment used \nresources.pg.host\n, \nresources.pg.username\n and \nresources.pg.password\n. \nIn our example, the chat component depends on the identity component, even though they are in different YAML files. To launch them with a single command, we use a special resource called a service that references a workload by name. This service adds the target workload's host and port to the environment, so one container can easily reach another.\nFor the namespace, we can see that pods (deployments) were created for each app (chat with 2 containers and identity with 1) and for each resource. We can also see a proxy that handles traffic redirection from the port-forward to each pod.\n\nThis proxy is created as an Nginx pod with randomised ports for each resource and service from the user YAML file. It sets up the namespace resources so the user can port-forward to them.\nPort forwarding\nAfter creating the XWorload resource, the HTTP request terminates with a success status code. At this point, the resources aren't created yet because the composition hasn't run.\nSo the CLI starts a loop that checks the workload status, with a sensible timeout.\nTo get this status, the CLI uses the workload name mentioned above. It sends a request to the server that identifies the namespace based on that workload name. With the Kubernetes API, it queries all resources in that namespace with the Crossplane label spec.crossplane.resourceRefs and returns their status to the CLI.\nWhen all resources are Running, then the CLI can try to port forward. To port forward, we rely on the nginx proxy that we saw before. The proxy is identified by name in that namespace, and each available port to port forward to is a port on that proxy pod. For each port, the CLI opens a connection and forwards traffic to it. \nLet’s recap:
⁠
⁠We specified what we want to create in the YAML files. As the example files show, the configuration is minimal (no Postgres version specified, no configuration environment variables, no resource or probe specifications). All of these resources were created in a remote cluster (not my local computer) and accessible via port forwarding.\nGenerating dynamic credentials\nCredentials for resources, like database passwords, are randomly generated during provisioning. To make them accessible to the user, we store them in Vault. The CLI includes a command to retrieve a dynamically generated credential from Vault in a single command. \n\nThis is done by configuring the \nVault database secrets engine\n, so when the eshu workload endpoint credential\n \ncommand runs, the CLI requests the control plane server, which uses the Vault API to generate new temporary credentials. \nFor Postgres, for example, we use the \npostgresql-database-plugin\n, and for each dynamic credential we create the user with:\n\nAs with other functionalities, we offer a unified interface. Resources without a Vault secrets engine still work the same way, but the credentials retrieved are static. In that case, it retrieves the credentials generated during resource provisioning.\nClean up of namespaces\nSince these resources are hosted in a remote cluster, we need to ensure they get cleaned up if the user never does it. To ensure that, we have a dedicated server that implements the Kubernetes type-reconciler interface and is associated with the created namespace. \nEach workload resource has an annotation with a target TTL. This server checks whether the namespace has already exceeded the target TTL. If so, it deletes the namespace and vault secrets. \nChallenges & benefits\nOne of the first challenges we ran into was password generation. Crossplane has a render cycle: every 60 seconds, the XWorkload is reprocessed, and any changes are reflected in the cluster. Since we are generating random information, for passwords, for example, this caused the environment to be outdated on the subsequent render cycle. In fact, anything that changed the desired resources on every cycle would cause the pods in the cluster to keep restarting. So it was important to ensure the function would eventually reach a steady state. For password generation, we handled that by checking the existing state of the resources and reusing the password if it was already present.\nAnother challenge was avoiding clashes when so many different resources are being created. We could force users to use different names for every port and container, but that would create a worse experience, especially when workloads are shared across teams. So instead, we decided to compose the names so they need to be unique only within the workload context. This quickly hit the 63-character limit for Kubernetes names. We fixed that by capping each part of the composed name at a specific character length.\nSo what benefits did we actually get from all of this, when compared with something like docker compose or test containers?\nFirst, these workloads are simpler to set up. The resources themselves require minimal configuration, and many defaults are quite sensible for most use cases. Especially when experimenting with a new technology that you might not be quite familiar with, this minimal configuration was helpful. \nAnother benefit is the unified resource interface. As mentioned earlier, you initialise database data via the same input attribute, even if the underlying database doesn't support it out of the box. This simplifies that process so that users can focus on what they are actually working on.\nAll the resources are also being launched in a remote cluster. In heavier apps, this makes quite a difference.\nLastly, all of these definitions are shareable, and a workload can import another. This lets teams more easily spin off full environments for their services. For now, this is a YAML file; in the future, we can build a centralised catalogue.

Automating Kubernetes Infrastructure with Crossplane Functions

In Part 1, Ivan (Principal Engineer with the Deposit & Savings team) walked through the three approaches that quietly power most scheduled jobs in production. This second part picks up where those simple solutions fail. \nHaven't read Part 1 yet? \nStart here\n\n_____________________________________\nWhen simple solutions are no longer enough\nSometimes you start a task in one direction, and halfway you just realise it's not working, especially because of scale. That's exactly what happened with us; as we grew, our product grew; deposits became far more numerous, which is great news. The bad news is that our single-instance processor could no longer cope. As always, it happened unexpectedly and under the banner of "well, who could have predicted that?"\nYet the symptoms are familiar:\nExecution time grows linearly with the data volume and stops fitting into the interval between ticks. The next run starts while the previous one hasn't finished. \nA heavy job blocks the node, and lightweight jobs sit idle. \nA crash mid-processing means redoing the work from the very beginning.\nWhere exactly it got stuck is hard to tell from the logs. \nAt this moment, it becomes visible that the job is actually two fundamentally different tasks packed into one method:\nFind the records that need to be processed\nProcess them\nIt makes sense to scale the second part. The first stays lightweight: even at large volumes, "find 100 thousand ids" is cheaper than "process each one." This separation is what opens up the space for everything discussed below.\nReader + Processor: scaling the processing\nReader → Kafka → consumers\nThe scheme:\n one instance selects work from a source (a database, an external API, anything) and publishes tasks to Kafka. Several consumers process the messages in parallel.\n\n\nPros\nNatural backpressure through consumer lag\nConsumers scale independently of the reader\nRetry and DLQ are standard practice. It's all been figured out\nFull separation of reading and processing; you can update and deploy them independently\nCons:\nKafka is an infrastructure you have to have and maintain\nOrdering only within a partition. If order by key matters, design the partitioning up front\nExactly-once delivery is unachievable in the general case; a message may arrive more than once (producer retry, consumer rebalance, redelivery after a failure). Therefore, the handler must be idempotent: reprocessing the same message must not change the result. That's what gives you exactly-once at the effect level, which is what people usually mean when they say "exactly-once." Kafka transactions cover the atomicity of "read → process → commit offset," but they don't relieve you of business-operation idempotency\nLag monitoring becomes a first-class concern\nReader → REST\nThe scheme:\n one instance selects work and calls the handler's REST endpoint in parallel, either a single one or several directly.\nPros\nNo Kafka, thus operationally simpler. \nSynchronous feedback: the reader immediately knows whether it was processed or not.\nDirect control over parallelism through the call-pool size. \nCons\nThe reader implements retry, timeout, and backpressure itself.\nA slow handler blocks a connection in the reader. \nNot suitable for large volumes. At scale, this turns into a homemade Kafka.\nNo natural replay/audit.\nThe shared problem of both approaches: the reader as a SPOF\nIn both the Kafka variant and the REST variant, the reader is a single instance with a latch (the same as ShedLock or leader election). If it crashes mid-batch-read, unpleasant questions arise:\nWhich records have already been sent for processing and which haven't?\nWill they be guaranteed to be resent on the next tick?\nWon't we send them twice?\nThese questions are solvable (idempotent reader, checkpoint after publishing, transactional outbox), but they haven't gone anywhere; they're simply hidden in the architecture. Next, we'll look at how to eliminate the reader-as-SPOF entirely.\nGetting rid of the reader as a SPOF\nDB-level partitioning\nThe main idea is to remove the reader as an explicit place in the architecture. Each worker pulls their own share of records directly from the database. There's no coordinator: the database serves as the arbiter.\nThere are two approaches, each with its own set of limitations.\nSELECT ... FOR UPDATE SKIP LOCKED\nIn PostgreSQL (since 9.5) and MySQL (since 8.0), you can atomically "grab" free records: the query locks the selected rows and skips the ones already locked by another transaction. Several workers run in parallel:\nSELECT * FROM jobs\nWHERE status = 'pending'\nLIMIT N\nFOR UPDATE SKIP LOCKED\nAnd each one gets its own non-overlapping set. Coordination is free, at the DBMS level. This is elegant, but there are two important "buts".\nDependence on the DBMS:\n FOR UPDATE SKIP LOCKED is supported in identical form by PostgreSQL (since 9.5), MySQL (since 8.0), and Oracle (since 11g — Oracle Advanced Queuing is built on exactly this). SQL Server assembles the same behaviour via the WITH (UPDLOCK, READPAST) hints; Google Spanner has no such mechanism at all; CockroachDB gained it relatively recently but with a number of caveats (column families, retries under SERIALIZABLE, contention). Before relying on it, check the current limitations for your version.\nProcessing must happen within a transaction:\n The lock is released on commit or rollback — while it's held, no one else will touch those records. If the processing is short and local, all is well. In our deposit example, this works while crediting interest is just an UPDATE to the balance. But as soon as, before crediting, you need to go to an AML service, fire a notification, or update the balance in a downstream system, keeping the transaction open during a network call is a bad idea: long transactions hit database performance, it's easy to catch a timeout, and it's easy to hang for a long time. This is usually cured by a two-phase pattern — a short transaction marks the records "in progress with worker X" and commits, then the processing happens outside the transaction, then a final transaction marks them "done." But that's no longer SKIP LOCKED in its pure form; it's a homemade queue on top of a table where SKIP LOCKED is used only at the grab stage.\nHash-based partitioning + Quartz\nAn approach that doesn't depend on the DB dialect and doesn't run into transaction length. The idea is to distribute records across shards in advance and bring up a separate reader for each shard.\nA shard_id column is added to the table, computed as hash({partition_key}) % N and saved when the record is inserted. In our deposit example, partition_key is, say, deposit_id — each deposit independently lands in its own shard. That is, sharding happens at write time, not at read time.\nThe architecture then looks like this:\nN triggers are configured in Quartz — one per reading service. A Quartz cluster is convenient in this role: it can distribute triggers across nodes and provide their failover.\nN reader deployments are rolled out in k8s, each with its own shard_id in the configuration.\nEach reader hooks onto its own trigger and independently reads only its own records (WHERE shard_id = X).\nFrom there, each reader works exactly as in the "reader → Kafka → consumers" scheme (see the "Reader + Processor" section): it publishes tasks to Kafka, and the consumers handle them\n\nIn practice, the number of shards is often taken to be larger than the number of readers — for example, 64 shards and 4 readers, each responsible for 16 shards. This gives headroom for scaling: adding new readers doesn't require recomputing shard_id.\nThe main virtue of this approach is the migration path.\n If a project already runs the classic Quartz + Kafka + handler scheme (that is, everything discussed in the "Simple solutions" and "Reader + Processor" sections), the transition is made without re-architecting: a shard_id column is added, the reader configuration is duplicated into N instances with different values, and N triggers are configured. The handlers (Kafka consumers) don't change at all. Evolution, not revolution.\nAmong the limitations:\nThe number of shards is chosen at the outset. Changing it later means recomputing shard_id across all existing records. That's why shards are usually taken with headroom\nWhen inserting a new record, you need to compute shard_id correctly. This is either computed in application code, or in a DB trigger, or the column is made generated/computed (if the DBMS supports it)\nN reader services mean N deployments, each with its own alerting and dashboard. In practice, this isn't scary (labels help), but the operational footprint is larger than with a single reader. This is partially mitigated by a k8s StatefulSet: instead of N separate deployments, a single object with N pods that have stable ordinal indexes (reader-0, reader-1, …), and shard_id maps directly onto that index. It's important to understand that a StatefulSet reduces the operational footprint but does not solve the problem below: if a pod crashes, k8s will restart it with the same index, and for the duration of the downtime the shard still sits idle\nThere's no rebalancing of shards across handlers. This is the main risk of the approach, and it's easy to miss behind the words "the Quartz cluster provides failover." Quartz will indeed pick up the trigger of a fallen node — but the shard → reader assignment is static; it's nailed to the pod's config. So if the handler of shard X itself goes down (a bad deploy, OOM, a crash loop), its trigger is alive and ticking, but there's no one to read WHERE shard_id = X: the other readers are bound to their own ranges and won't take someone else's. Shard X sits idle until the pod recovers. The Quartz cluster protects triggers, but not handlers — and here it's precisely the handler that fails\nWhat both approaches have in common\nThe reader as a SPOF either disappears or is broken up into N independent parts. In SKIP LOCKED — through workers grabbing records directly, without a dedicated reader. In hash + Quartz — through the readers becoming N, where the crash of one paralyses only its shard (1/N of the total work), and the Quartz cluster guarantees that another node picks up that shard's trigger.\nIf an idle shard is an unacceptable risk, rebalancing has to be added separately — by hand or with a ready-made tool. The DIY path moves shard assignment into a coordinator backed by ZooKeeper or etcd: each reader registers an ephemeral record on startup ("I'm alive, I hold these shards"), and when a pod crashes, its record expires, a watch fires on the others, and the fallen shards are reassigned to the living. Shard → reader becomes dynamic rather than configurational. Apache Curator gives the primitives (LeaderLatch / LeaderSelector for electing a coordinator), but the shard-distribution logic on top you write yourself.\nThe ready-made path is Apache ShardingSphere ElasticJob (Lite) — the same combination under the hood (Quartz for triggers + ZooKeeper for coordination), but with sharding and rebalancing already built in: slices are redistributed automatically when a node is added or removed, and an optional failover finishes off an interrupted slice within the current cycle. The price is a hard ZooKeeper dependency and lock-in into the framework's Job API, which loses the seamless "Quartz + Kafka" migration this whole approach was built for. Worth a sober look at the project's state, too: the core is barely developed (last functional release end of 2023), and ZooKeeper coordination can itself become a point of failure under network instability. As proof, the problem is solvable out of the box; it's useful; as a default for a new project, debatable.\nSpring Batch remote partitioning\nSpring Batch is a framework purpose-built for batch processing: reading in batches, chunk-oriented transactionality, retry/skip policies, persistent job state in a JobRepository, restart capability "from the point where you fell." For our task, one specific capability is interesting — remote partitioning.\nThe idea is simple but important: \nthe manager doesn't read the data. The manager only describes how to split it.\n Concretely, it works like this:\nThe job has a Partitioner, a component that produces N "execution contexts." Each context is a dictionary of parameters, for example, {minId: 0, maxId: 9999}, {minId: 10000, maxId: 19999}, and so on.\nThe manager dispatches these contexts to the workers through a broker (Kafka, RabbitMQ, JMS; Spring Integration can work with any of them).\nThe workers listen to the queue. Each one takes a single context and launches at home a step whose ItemReader is parameterised by these minId/maxId; that means it pulls only its own range from the database.\nThe worker processes its partition fully (read + process + write) and sends a response to the manager.\nThe manager waits for responses from all partitions and closes the job\nThe fundamental difference from "reader → Kafka" (see the "Reader + Processor" section): \nthe data doesn't fly over the network.\n Only the metadata "go read this range" flies over the network. The records themselves are pulled by each worker directly from the database.\nWhat this gives you:\nThe reader as a SPOF disappears: the manager does only cheap work; slicing the ranges. The reading of data is distributed.\nRestart at the partition level: 2 of 10 workers failed on retry. Only those 2 restart, not the whole job.\nDeclarative scaling: change gridSize from 4 to 16, so get more partitions without rewriting code.\nThe state of each partition is tracked persistently in the JobRepository: you can see what failed, where, and on what, rather than "it started, then it finished" from the @Scheduled world.\nWhat hurts:\nThe learning curve is steep. Job, Step, JobLauncher, JobRepository, Partitioner, StepExecutionAggregator, PartitionHandler. There are many concepts, and without some time to get up to speed, you can't work productively with the framework.\nIts own tables in the database. The story is the same as with Quartz — Spring Batch is also stateful, with a dozen service tables.\nA skewed partitioning function is the main pain in practice. If you split by ID ranges, but the actual records are dense in one range and sparse in another, one worker hauls 80% of the work while the rest sit idle. Good partitioning is a separate skill: range, hash, time-based, custom; and it's tuned to the specific data profile\nThe manager-worker round-trip adds latency. For short jobs (seconds), the overhead can exceed the benefit. Spring Batch makes sense where it's about minutes and hours of work\nNot for streams. This is about "run through a finite set and process it," not about continuous processing of incoming events\nThis is justified when one-off or periodic runs over a large finite set, where restart capability, an audit trail of "what got processed," and partitioning at the level of the task itself rather than the infrastructure beneath it matter.\nDurable execution: Temporal and its neighbours\nEverything discussed above lives inside the JVM world and k8s primitives. But there's a separate class of tools worth keeping in mind — external orchestrators with durable execution: Temporal (and its ancestor Cadence), and partly Airflow and Argo Workflows for batch scenarios.\nWhy is this relevant to our example specifically? As soon as crediting interest stops being a single UPDATE balance and turns into a chain of "check in AML → credit → update the balance downstream → send a notification," where each step is a network call that may crash, hang, or return more than once, we start reproducing by hand exactly what Temporal gives out of the box:\nDurable execution rate: \nThe workflow remembers which step it stopped at. It crashed on the notification after a successful crediting — on retry. Temporal won't credit again; it'll continue from the notification. This removes a significant part of the "don't do it twice" pain, which in DB-centric schemes you have to close with idempotency keys and an outbox by hand.\nRetry as a declaration, not as code.\n The retry policy (backoff, limits, not retrying certain errors) is set by the step's configuration, rather than being written into every handler.\nVisibility.\n The state of each workflow instance is tracked and viewable — this closes that very "where exactly it got stuck" pain we started with.\nWhat we pay\n: a separate piece of infrastructure (a Temporal cluster or Temporal Cloud), its own programming model (deterministic workflow code, separated from the activities), and a noticeable learning curve. For a simple periodic job, this is a cannon on sparrows — no less so than Spring Batch. In many teams, adding yet another heavy infrastructure component is more expensive than squeezing out a solution on what already exists (a database, Kafka, Quartz) — which is why the rest of this article stays in the JVM world. But if the core of the task is reliably running multi-step processes with external calls and a strict "exactly once by effect," it's worth looking toward durable execution earlier than erecting a third layer of homemade orchestration on top of a table.\n\nConclusion\nIn system design, there's no silver bullet. The same approach will turn out elegant in one project and over-engineered in another.\n@Scheduled + ShedLock, a Quartz cluster, or a K8s CronJob cover a huge share of real-world tasks — and dragging Spring Batch in there would be shooting sparrows with a cannon. Where the infrastructure is controllable, you can lean on the capabilities of a specific database: SKIP LOCKED in Postgres or MySQL gives almost-free coordination of workers. And where you need to work with a distributed database or with heavy processing involving external calls, DB-agnostic solutions are needed: hash partitioning with Quartz, Spring Batch with remote partitioning, something else.\nSeparately, it's worth paying back the debt on observability.\nIn this article, I've covered only some of the possible options and tried, for each, to indicate where it works well and where it breaks down. The main skill in this area isn't "knowing the single right approach," but matching the characteristics of the task to the characteristics of the solutions and choosing the least bad one for the specific context.\n________________________________________\nWe're looking for engineers like Ivan, who take their time to understand a system deeply enough to know exactly when to stop reaching out for more architecture because the problem in front of them doesn't need more architecture. If you're that engineer, we'd love to speak to you. \n\nExplore open engineering roles at Moniepoint\n. \n


Scheduled jobs in Java + Spring + Kubernetes - Part 2

Before we get into the article, let's meet Ivan. \nIvan is one of our Principal Engineers, and he partners with product managers, software developers, QA engineers, etc to deliver a seamless experience for our savings feature customers. He has spent over 10 years building high-load backend systems across fintech, e-commerce and analytics.  \nIn this article series, he takes us through his experience with scheduled jobs in Java; from a single scheduled method to sharded, fault-tolerant processing across millions of records \n____________________________________________\nThere's a phrase in the running example I can't pass over in silence — "don't credit twice." It's an important problem, but it's a problem of processing idempotency, not of scaling jobs, and I won't be solving it here. The one point worth fixing so it doesn't hang in the air: no scheduler by itself guarantees that a business effect is applied exactly once. ShedLock, Quartz, SKIP LOCKED, Kafka; they all reduce the probability of a double trigger, but "exactly once by effect" is a property of the data model (idempotency keys, transactional boundaries, outbox), not of the schedule. Everything below is about making the work reach execution and scale, not about its financial correctness.\nSimple solutions\nBefore going off into complex architecture, let's honestly look at what usually works.\n@Scheduled + ShedLock\nI'd call this approach not a "scheduler" but a "latch on top of a scheduler." Spring already knows how to run annotated methods on a schedule via @Scheduled. The problem arises when the service is deployed across several instances: each one will dutifully run the method at the tick, and the same job gets processed N times. ShedLock solves exactly this. It takes a lock in a shared store (a database, Redis, Hazelcast, etc.) and guarantees that only one instance runs.\nIn practice, it's worth remembering two settings that confuse everyone:\nlockAtMostFor (the maximum lock-hold time):\n Insurance against a deadlock. If a node dies while holding the lock, the others can proceed only after a time has elapsed.\nlockAtLeastFor (the minimum lock-hold time): \nProtects against a fast job finishing in a millisecond, releasing the lock, and a second instance immediately re-running it.\nWhen it fits\nRare, lightweight jobs in a service that already exists, not requiring run history or complex recovery.\nWhat doesn't work\nIt's always single-instance execution; it doesn't scale horizontally.\nNo misfire policy: if all instances were down at the tick, the run is simply skipped; no one will catch up.\nThe schedule lives in code or config; any change means a deploy.\nNo built-in run history or complex retry\nSpring + Quartz in clustered mode\nThe main idea about Quartz that's often missed: it distributes different jobs across nodes; it doesn't parallelise a single one. If you have 10 different jobs and 3 nodes, Quartz will spread the load nicely. If you have one big job, Quartz won't make it any faster.\nWhat Quartz gives you\nA persistent job store with run history.\nMisfire policies; handling of missed ticks.\nDynamic triggers that can be created and changed from code.\nCalendars, complex cron expressions, custom triggers\nMisfire policies, by the way, are treacherous, and the behaviour depends on the trigger type. For a SimpleTrigger, an instruction like MISFIRE_INSTRUCTION_FIRE_NOW / ...RESCHEDULE_NOW_WITH_EXISTING_REPEAT_COUNT after a multi-hour outage can make the job "catch up" with the schedule in a burst — and that's most likely not the load spike you expected. For a CronTrigger, the logic is gentler: most often, a single immediate fire happens, and then it's back to cron. The practical takeaway: before choosing a misfire instruction, look at which trigger you have and what exactly that instruction does in its terms. The defaults for Simple and Cron are different.\nAnd a separate, real-life practical trap: if the beta and prod environments point at the same database (which happens more often than one would like), you need to watch very carefully that Quartz is turned off on beta. Otherwise, both environments will see the same jobs in the Quartz tables and compete for them — beta instances will start regularly snatching prod tasks. Debugging pain here is guaranteed, and it's all the worse because the symptom is flaky: now prod "didn't run," now beta suddenly did something to prod data.\nWhen it fits\n \nMany heterogeneous jobs, you need history, you need misfire policies, you need to change the schedule from code without a redeploy.\nWhat doesn't work\nOne job = one node at a time. There's no parallelism inside a job.\nA full set of Quartz tables in the database, constant polling.\nKubernetes CronJob\nA K8s CronJob isn't a way to "run code on a schedule." It's a way to run code in isolation. Each firing spins up a separate pod with its own resource limits, its own lifecycle, its own logs. A job crashing won't drag down the main service, and a heavy nightly aggregation won't eat the resources allocated to serving traffic.\nSubtleties that are often forgotten\nConcurrency Policy: What to do if the previous run hasn't finished yet (Allow / Forbid / Replace).\nStarting Deadline Seconds: If the controller missed a tick, how to wait before attempting the run. A separate footgun: if you leave it empty and the controller, for some reason, misses more than 100 schedules in a row (for example, kube-controller-manager was down for a while), the CronJob stops launching altogether and cannot determine if the job needs to be started. Too many missed start times show up in the events. The cure is setting a reasonable startingDeadlineSeconds so the "missed" window isn't counted from the beginning of time.\nA cold start of a Spring Boot pod is realistically 20–40 seconds. For a short job, that's half its working time\nWhen it fits\nHeavy jobs that need resource isolation; tasks with explicit memory/CPU requirements; teams comfortable living in k8s.\nWhat doesn't work\nEach job is a separate entity in CI/CD. Ten jobs = ten manifests.\nObservability is fragmented: logs are scattered across a heap of short-lived pods.\nEither a separate image, or the same one with a different entrypoint/profile — you need to decide at the outset.\nA decision matrix\nThe selection logic boils down to two questions:\n\nThe first question rules out cases where the job needs separate resource limits or isolation from serving traffic; that's K8S CronJob territory. The second filters for situations where Quartz genuinely pays off: many heterogeneous jobs, you need run history, misfire policies matter, or a dynamic schedule. If neither is about your task, @Scheduled + ShedLock cover it without extra fuss.\n____________________\nThe above are simple solutions that work when you're servicing a small number of users or cases. What happens when your scale grows? In Part 2, Ivan shares what he then does when the simple solutions are no longer enough for the scale we're building at. 

Scheduled jobs in Java + Spring + Kubernetes - Part 1

Most infrastructure stories get written after something breaks. This one is different. Nobody paged Celestina Amadi's team at 3 AM about Kafka. The VMs were working, yet she made the call to rebuild anyway because she could see where things were headed before they became a problem.\nCelestina Amadi leads the Cloud Engineering team behind the infrastructure that powers Moniepoint's payment and savings products, building systems that move transaction and savings data reliably for millions of customers every day. She's also a Grafana Champion, a HashiCorp Ambassador, and an IBM Champion 2026.\n━━━━━━━━━━\nWhy We Moved to Strimzi\nIt Wasn't a Crisis. It Was a Decision.\nWe moved to Strimzi because we looked honestly at how we were running \nKafka\n and made a deliberate call: this does not scale, and we can do better.\nThat kind of decision is actually harder than reacting to a crisis. Incidents are obvious — something breaks, and you fix it. Proactive architectural change requires you to see clearly enough to act on discomfort before it becomes a disaster. It requires you to say "this is working, but not well enough" and mean it.\nThis is the story of what we saw, what we built, and what it taught us about thinking in systems.\n\nHow We Were Running Kafka Before\nOur Kafka setup started the way most things do in a fast-moving engineering team: pragmatically.\nWe needed CDC pipelines. Moving data from Database A to Kafka to Database B, from Database C to Kafka to Database D. Multiple pipelines, each serving a different business flow. We spun up Kafka instances on VMs, managed with Docker Compose as needed. They worked. We moved on.\n\nOver time, the seams started showing.\nEvery change required SSH and port-forwarding. Our Kafka instances had no URL. You accessed them via localhost. To make any change, whether adding a source connector, adding a sink, or updating a configuration, you had to SSH into the server, port-forward to get local access, and make the change manually. Every time. In every instance.\nWith Docker Compose, downtime was always one command away. Managing Kafka with Docker Compose works until it doesn't. Any update or fix that required restarting the stack meant running `docker compose restart` and hoping the restart went cleanly. For a data pipeline, that's not a comfortable position to be in.\nVersions were going stale, and updates were per instance. Kafka evolves. New version released. But because each instance was a separate VM managed independently, upgrading meant going into each one individually. To make it worse, our instances weren't even consistent. Some ran on ZooKeeper, some had adopted Raft as Kafka evolved. Different consensus models, different operational behaviour, different things to know before touching anything.\nProvisioning a new pipeline meant reinventing the wheel. No standard process. No shared configuration. Every new CDC pipeline was a fresh set of decisions: which version, which consensus model, which configuration. Knowledge of how each cluster worked lived with whoever set it up.\nMonitoring was manual.  We used JMX Exporter to pull metrics and eventually shipped logs to Loki. These helped. But they were layers on top of an operational model that still required you to SSH into the right instance to actually diagnose anything. There was no unified view of the fleet.\nNone of this was broken in a way that triggered an alarm. It was just slow, manual, and increasingly inconsistent as the fleet grew. We were spending engineering time on operations that should have been automatic.\n\nWhat Is Strimzi?\nBefore getting into how we implemented it, it helps to understand the core idea.\nStrimzi is Kafka running on a Kubernetes cluster, managed by a \nKubernetes operator\n. An operator is a piece of software that runs inside your cluster and continuously manages another application on your behalf. It watches a set of configuration files and ensures the actual state of your Kafka deployment always matches what you declared. If something drifts, the operator corrects it. If a broker goes down, the operator brings it back.\nThe key shift: you stop being the operator. The software becomes the operator.\n\nStrimzi\n is open-source and part of the \nCNCF\n — no licensing costs, no vendor lock-in. Everything is expressed through Kubernetes custom resources (CRDs), which means your entire Kafka setup is code that lives in Git.\n\nHow We Implemented It \n\nWe didn't just drop Strimzi onto an existing cluster and call it done. We made deliberate decisions about how to structure it.\nDedicated Kubernetes cluster:\n We created a dedicated cluster for our Kafka workloads on GCP/GKE. Separating Kafka onto its own cluster gave us clean resource boundaries and made it easier to manage capacity independently from our application workloads.\nNamespace isolation per pipeline:\n Within that cluster, each Kafka deployment lives in its own namespace. The pipeline from DB A to DB B gets its own namespace. DB C to DB D gets another. Instead of "one VM per pipeline," we now have "one namespace per pipeline" — with real isolation between them. A misconfiguration in one namespace cannot consume resources or affect availability in another.\nYAML templates as the standard: \nWe created standardised Strimzi cluster templates. When a new pipeline needs a Kafka instance, you take the template, fill in the relevant fields, and apply it. Each template covers the full stack: the Kafka cluster definition, Kafka Connect, the node pool, and metrics configuration. Every new instance comes out consistently because it comes from the same starting point.\n\nArgoCD\n for sources, sinks, and configuration:\n Every connector (source, sink, configuration) is defined in a YAML file and lives in a Git repository. To add a new source or sink, you write a YAML file and open a merge request. Once it's merged, ArgoCD picks up the change and automatically syncs it to the cluster. The operator applies it from there.\nCustom Kafka Connect image as a single source of truth:\n We extend the official Strimzi Kafka image and bake in all the connectors we use. That includes Debezium connectors for CDC (PostgreSQL, MySQL, and Spanner) plus ClickHouse, MongoDB, S3, and the Confluent JDBC connector with the relevant JDBC drivers. All connector versions are pinned explicitly in the Dockerfile. When a connector needs an upgrade, we update a single line in the Dockerfile and build a new image. When a new Kafka version is released, we rebase onto the updated Strimzi image. One place. One PR. The operator rolls it out across all clusters.\n\nWhat Changed Day-to-Day\nThe difference in day-to-day operations was immediate.\nNo more SSH:\n No more port-forwarding. To add a source or sink, you write YAML and push. The operator applies it. What used to require terminal access to a VM is now a pull request that gets reviewed and merged. \nNo more Docker Compose restarts:\n Strimzi manages the broker lifecycle. You don't `docker compose down` a production data pipeline anymore.\nScaling is a number in a file\n: Need an additional broker? Change one value in your YAML. The operator handles placement and ensures the cluster state stays consistent. Need to scale down? Same. It handles graceful decommissioning.\nLogging without the archaeology:\n Every broker's logs are available via standard Kubernetes tooling — `kubectl logs` from the terminal, or directly in Lens and k9s if you prefer a UI. We still ship to Loki for retention and cross-cluster querying.\nA single UI that shows everything at once:\n We use Kafbat internally, which gives us a unified view across all clusters, including broker health, connector status, consumer lag, and topic metadata. When a connector fails, you can see why it failed, inspect the configuration, and understand the issue from a browser. No terminal required.\nRolling updates without downtime:\n Kafka upgrades happen one broker at a time, with the operator ensuring quorum is maintained throughout. Most of our cluster updates now happen without downstream systems noticing.\nSelf-healing:\n If a broker crashes, the operator restarts it. If Kubernetes can't schedule it on the current node, it reschedules elsewhere. You get an alert that something happened and was resolved — not a page asking you to fix it manually at an inconvenient hour.\n\nA Few More Reasons Worth Naming\nThe narrative above covers most of it, but three points don't show up in the day-to-day story.\nCost efficiency:\n Strimzi is open-source with no licensing fees. Running Kafka inside the same Kubernetes cluster as your consumers and producers also cuts network egress costs you'd otherwise pay for traffic crossing VPC boundaries to reach an external Kafka endpoint.\nLower latency:\n When producers and consumers live in the same cluster as Kafka, the network path between them is short. External setups add hops. For high-throughput CDC pipelines, that difference is measurable.\nEnvironmental parity: \nBecause your entire setup is YAML in Git and synced by ArgoCD, cloning it from production to staging is straightforward — fill in a template, point it at a different namespace, and apply. The same configuration that runs in production can run locally in Minikube for debugging.\nThis was the right call for us, but it's not universal.\nStrimzi is worth it if:\nYou're already running Kubernetes, or seriously committed to it\nYou're managing multiple Kafka pipelines with inconsistent configurations\nYour team is spending real time on SSH access, manual connector management, or per-instance version updates\nYou want GitOps as a genuine practice — not just a concept\nYou have the Kubernetes maturity to manage operators safely\nStrimzi probably isn't the right fit if:\nKafka is peripheral to your product. Use managed Confluent Cloud or MSK instead\nYour organisation isn't comfortable with Kubernetes yet\nYou have a single, simple, well-tuned cluster that isn't causing problems\nYou need commercial support SLAs. Strimzi is community-backed, not commercially supported\nYou want pre-built integrations for monitoring, alerting, or schema registries out of the box. With Strimzi, you wire those up yourself\nFor us, the signals were clear: multiple pipelines running inconsistently, SSH and Docker Compose eating engineering time, no unified visibility, and an existing Kubernetes investment. The only real question was why we hadn't done it sooner.\n\nThe Lesson\nYou can't think in systems when you're drowning in operations.
Looking back, the biggest thing running Kafka on VMs taught us wasn't about Kafka. It was about the manual infrastructure costs that you can't easily measure.\nEach individual task was manageable. But collectively, the manual work consumed the cognitive space that should have been available for thinking about the system as a whole. We were so busy managing the current state of things that we never stepped back to design how they should work.\nMoving to Strimzi didn't just reduce our operational burden; it changed what our team was able to think about.\nWhen the infrastructure is code, you can reason about it. When operations are declarative, you can predict them. When complexity is visible in Kafbat instead of buried across disconnected instances, you can make architecture decisions rather than just firefighting.\nNew engineers can read the YAML and understand the entire Kafka setup. Changes are reviewed before they happen. The system tells you when something goes wrong and often fixes itself before you get involved.\nThat's the shift that matters: from operators who keep things running to engineers who design systems that run themselves.\n━━━━━━━━━━\nDecisions like the one in this article don't come from chasing incidents. They come from a team with the standards to ask "is this actually good enough?" before something forces the question.\nIf you'd love to build systems that run themselves rather than spend your career keeping systems from falling over, we're looking for you.\n\nExplore open engineering roles at Moniepoint\n.\n


What Running Kafka on VMs Taught Us About Systems Thinking

Every engineering team right now is having some version of the same conversation: AI can write the code, so what's left for the engineer to do? Most of the takes you'll read are either breathless ("10x your team overnight") or defensive ("it's just autocomplete"). Neither is useful if you're actually responsible for a system in production.\nThis piece, by Pavan Venkatesan, argues something less comfortable: that AI hasn't made engineering as a whole easier; rather, the thinking part of engineering has become harder and is now the only part that still matters. A pull request can look immaculate and still not account for how pod affinity rules can starve an availability zone under real traffic. Polish is no longer proof of understanding.\nIf you're wondering what actually separates a strong engineer from a fast prompt writer right now, this is worth five minutes of your time. \nIntroduction\nFor decades, credibility in software came from what you built with your own hands. Writing robust systems required stamina, coordination, and deep technical craft. The effort itself acted as a filter; only ideas worth the cost were implemented.\nThat constraint is gone.\nToday, large language models can generate entire services, documentation, tests, and deployment scripts in minutes. The barrier to producing software artefacts has collapsed. The limiting factor is no longer typing speed or knowledge of syntax; it is clarity of thinking.\nInside an organisation adopting AI, this shift is profound. A team can spin up a prototype payments reconciliation service over a weekend using AI assistance. But the real question becomes: Who defined the problem well? Who validated the trade-offs? Who understands the system well enough to maintain it?\nCode is abundant. Judgment must not become scarce.\nWhen Output Becomes Cheap, Signal Changes\nWell-structured codebases and thorough documentation used to be signals of maturity and care. Today, AI tools can generate pristine README files, polished APIs, and layered architectures instantly.\nThis makes surface quality an unreliable proxy for depth.\nWithin an AI-enabled engineering team, you might see beautiful documentation generated in seconds, clean abstractions suggested by the model, and automated test scaffolding created in bulk, but none of these guarantee that the system handles scale, edge cases, or operational realities. \nAn AI can generate an event-driven order processing service, complete with retry logic, dead-letter queues, and idempotency keys, and the code will look immaculate. But does it handle message ordering under partition rebalancing? Does the retry backoff account for downstream rate limits? A model can scaffold a Kubernetes deployment manifest with health checks, resource quotas, and horizontal pod autoscaling,  yet miss that the pod affinity rules will starve a specific availability zone under real traffic patterns.\nThe signal of quality shifts from \nhow polished it looks\n to something harder to fake: Does the team understand the system deeply? Is there ownership and accountability? Can someone explain why the circuit breaker threshold is set to 60% and not 40%, without re-prompting the model?\nIn AI adoption, governance and provenance become more valuable than aesthetics.\nThe Compression of Effort\nAI drastically reduces the mechanical cost of building software. Tasks that once required weeks, writing gRPC service definitions with protobuf schemas, building CDC pipelines to sync state across bounded contexts, and standing up end-to-end integration test harnesses with containerised dependencies, can now be compressed into hours.\nThis is not hype. It is leverage.\nUsed correctly, AI frees engineers to spend more time on architecture and systems thinking, explore multiple design alternatives quickly, and run experiments that previously felt too expensive. In an organisation adopting AI at scale, this means faster iteration cycles and broader experimentation: scaffolding a complete OAuth2 flow with PKCE, token rotation, and role-based access control in an afternoon instead of a sprint; generating contract tests between fifteen microservices to catch breaking changes before deployment; spinning up a feature-flagged canary release pipeline to compare two ranking algorithms in production traffic. Teams can prototype an entire event-sourced domain model, evaluate its query performance against a traditional CRUD approach, and make an informed architectural decision, all before the old process would have finished the design document.\nHowever, speed without discipline produces fragile systems. The advantage goes to teams that combine AI acceleration with experienced technical oversight.\nThe Risk: Infinite Artefacts, Finite Understanding\nIf software can be generated endlessly, its perceived value drops. A pull request that took weeks of focused effort once carried an implicit sense of respect. Now, large AI-generated diffs may appear instantly with unclear human involvement.\nThis creates tension. Reviewing becomes harder than generating. Accountability becomes blurry. Junior engineers may rely on tools without building fundamentals.\nFor organisations, the real danger is not "bad AI code." It is teams that lose the ability to reason about systems independently.\nIf AI writes the data pipeline, optimises the SQL, and configures the infrastructure, but no one fully understands the flow, technical debt becomes invisible.\nAI adoption must therefore include a strong review culture, explicit learning paths, and clear architectural ownership. Otherwise, productivity gains today become fragility tomorrow.\nTalk Becomes the Multiplier\nIn this new landscape, the most valuable skill is not typing code; it is articulating intent.\nThe engineer who can frame a problem precisely, define constraints clearly, ask the right questions, and evaluate model output critically will outperform someone who merely executes.\nAI does not eliminate engineering roles. It amplifies differences in clarity and systems thinking. Organisational structures compress too - design, coding, testing, and iteration increasingly blur into tight feedback loops.\nThe Future Is Thinking-First\nThe future of software development is not code-first. It is thinking-first.\nWhen machines can generate implementation at scale, the competitive edge shifts to strategic problem definition, technical stewardship, responsible governance, and mentorship that builds foundational skills.\nCode may be cheap now. But reasoning, accountability, and leadership are not.\nWe're looking for engineers who treat AI as leverage, not as a replacement for judgment: people who can frame a problem precisely, push back on a model's output, and own a system well enough to explain it without re-prompting anything.\nIf that sounds like how you already think, we'd like to talk.\nExplore open engineering roles at \nMoniepoint\n.\n



When Code Becomes Cheap, Thinking Becomes Expensive

Every session on Moniepoint's Personal and Business Banking products (whether it starts on Web, Mobile, POS, or USSD) begins with a call to a service that my team is responsible for keeping up. Being at the front door for over 10 million businesses and individuals, "high availability" stops being an architecture diagram with a green checkmark on it and becomes a number you owe, every single day.\nIn this post, I show the actual math of how Mean Time to Detect (MTTD) and Mean Time TO resolve (MTTR) get measured, why default Spring Boot health checks lie to you in production, and how connection pool sizes and circuit breaker thresholds get derived from real throughput numbers. \nMeasuring Availability and Scalability\nTwo properties define whether a platform survives production:\nAvailability\n is the percentage of time a system is operational and accessible to users.\nScalability\n is the system's ability to handle increasing load (more transactions, more users, more data) without performance degradation.\nFor a platform serving over 10 million businesses and individuals, a 0.1% availability dip is not a line on a dashboard. It is thousands of transactions that did not go through, and thousands of customers who lost trust in that moment.\nHow we measure availability\nWe target \n99.99% uptime\n on all critical services. To measure this, we register every microservice health check in \nGCP Uptime Checks.\n Each service exposes a health endpoint, GCP Uptime polls it from multiple regions, and we alert on any sustained failure.\nFrom this, we derive two numbers that drive every incident review:\nMTTD (Mean Time to Detect):\n How quickly our monitors catch an anomaly before a customer does.\nMTTR (Mean Time to Resolve):\n How fast we restore service once an incident is detected.\n\nFigure 1: Uptime trajectories for a Tier 1 microservice.\nTip: Default Spring Boot Actuator health checks are unpredictable in production. They pull in dependencies (DB, cache, message broker) whose transient failures can unnecessarily flap a service out of rotation. We always implement a custom health check we fully control, so we can fine-tune the logic separately for startup, readiness, and liveness probes.\nHow we measure scalability\nScalability is harder to measure passively. We measure it by running load tests against expected and projected traffic, and tracking latency (p95, p99) and error rate as throughput increases. A service is "scalable" for us when we can show the latency curve stays flat up to 2x our peak traffic.\n\nFigure 2: Latency consistency relative to increased throughput.\nKey Principles\nAchieving high availability and scalability starts from the design. A poorly architected and configured system will cause sleepless nights in production. In order to achieve high availability, we adhere to the key principles below:\n1. Build For Failure\nServices must be stateless and horizontally scalable, and observability (metrics, logs, alerts) must be live at launch rather than added later. Failure modes are a design input, so we define expected failure scenarios during the RFC stage and build the metrics, logs, and alerts to cover them before the service ships.\n2. Configure For Scale\nDefault configurations will not survive real throughput. Every team documents the non-functional requirements of their service (expected throughput, latency SLOs), and sizes every queue, connection pool, thread pool, and timeout from back-of-envelope calculations grounded in those numbers. A useful reference on this is\n \nQueuing Theory for Software Engineers\n; you can also check out this video by our VP of Research & Development: \nMoniepoint Research Talks 7.0: The Mathematics of Software Engineering\n.\n\nFigure 3: Configuration for Redis connection pool.\n3. Apply Resiliency Patterns\nA retry storm will take down a healthy service faster than the original incident. Every inter-service call on our platform uses circuit breakers to prevent cascading failures, exponential backoff with jitter on retries, and fallbacks when a dependency degrades. Applying the patterns is only half the job; they must also be configured properly. The same discipline from Principle 2 applies: circuit breaker thresholds, retry counts, backoff windows, and fallback timeouts all need to be sized from the service's real throughput and latency profile, not left at library defaults.\n\nFigure 4: Circuit Breaker configuration for a dependency.\n4. Prove It\nPerformance claims need evidence, and intuition is not evidence. Every non-trivial change must pass automated tests (unit, integration, and E2E) before being merged, and every new service must pass a documented load test before going to production.\n\nFigure 5: Analysis of load test outcomes for a microservice.\nConclusion\nBuilding highly available and scalable systems is difficult, and the target moves every quarter. The principles do not. Measure everything. Design before you build. Size your configs to the real load. Defend against retry storms. Contain failures when they happen.\nOur goal is simple: deliver a seamless experience to over 10 million businesses and individuals, and hit 99.99% availability while doing it. We are constantly learning, constantly measuring, and constantly refining the way we build.\nIf you'd like to help us meet this goal, check our \ncareers page\n for open roles. 

High Availability in Production: What Running at Scale Actually Requires

If you use the Moniepoint savings feature, I bet you're familiar with the Save-As-You-Transact (SAYT) feature. The article below is a note from the team that engineered this life-saving feature. John, the writer, is a Head of Engineering at Moniepoint, leading multiple teams, and as you guessed, the savings team is one of those. \nAs the Head of Engineering for Savings, POS services and Accounts & Payments, he spends most of his time in meetings, reviewing decisions made by his engineering managers, assessing risks, and ensuring the architecture his teams are pushing exceeds customer expectations.  \n____________________________________________________\nWhat SAYT does, in one paragraph\nSAYT lets a customer set a rule, such as "save N500 every time I make a transfer" or "save 5% of every card transaction." Once set, spending triggers savings as a by-product. Every time they transact through any of our channels, the configured amount or percentage moves from their main account into a SAYT wallet. And with this feature, customers can withdraw when they want. \nThat's the product. The interesting half of this article is the engineering required to make it feel like a single, simple promise to the customer, when, in reality, it sits atop millions of events flowing through a distributed system every minute.\nThe architecture, at a glance\n\nFigure 1: SAYT producer-bus-consumer architecture\nProducer services emit events when transactions complete. The event bus stores them in a durable, partitioned log. SAYT consumes them, decides what matters, and hands off the work that actually moves money. Nobody calls SAYT directly, and SAYT doesn't call its own internals directly either. Even the hop from the event service to the core service goes through the bus.\nWhy didn't we just integrate directly?\nThe intuitive approach is for every transactional service to call SAYT after it finishes its work. It works in a demo and collapses in production. Every producer team ends up with SAYT in its dependency graph, having to handle SAYT being down, slow, or in the middle of an API change. New transaction types only flow into SAYT if someone remembers to wire them in. SAYT, which was meant to be a small thing on the side, becomes part of the critical path of every transaction.\nSo we inverted it. Producers publish events. SAYT subscribes. The bus is the contract between them. The event itself is small and structured:\n{ "event_id": "evt_01HX7K...",\n  "customer_id": "cus_8a4f...",\n  "event_type": "transfer.completed",\n  "amount_minor": 250000,\n  "currency": "NGN",\n  "produced_at": "2026-05-05T09:14:22.118Z"}\nIt's a statement of fact. Anyone who cares can listen. Anyone who doesn't can ignore it.\nWe split SAYT into two services on purpose:\nThe event service is CPU-bound and lookup-heavy. It reads events, checks matches, and either drops them or publishes a credit instruction to an internal SAYT topic.\nThe core service is database-bound and transaction-heavy. It consumes that internal topic, evaluates rules, and credits the customer's SAYT pot.\nTwo things to notice here. First, those workloads scale on different curves. Second, and more importantly, the hand-off between them is \nalso\n event-driven. The event service does not call the core service over HTTP. It publishes to a topic that the core service consumes at its own pace.\nThis is a deliberate backpressure decision. The event service runs hot. If we let it call the core service synchronously, a sudden spike on the bus would translate directly into a flood of HTTP requests to a service that does transactional writes against a finite-throughput store. With an event-driven hand-off, the bus absorbs the spike. Credit instructions accumulate on the topic. The core service drains them at the rate its database can sustain. The general rule we've internalised: any time a service produces work much faster than another service consumes it, the hand-off should be a topic, not a call.\nScalability: why event-driven, and why horizontal\nThe argument that mattered for picking event-driven wasn't elegance. It was \nhorizontal\n scalability.\nVertical scaling has a hard ceiling: there's the biggest available instance, and once you hit it you have nowhere to go. Horizontal scaling has none. You add instances as load grows, and the loss of any single one is absorbed by the others.\nEvent-driven architecture is what makes this work at our volume. The bus splits each topic into a fixed number of partitions. Consumers join a consumer group, and the bus assigns each partition to exactly one consumer in the group at a time. Add a consumer, and the bus rebalances. Lose a consumer, and the bus reassigns. The whole thing is elastic, with no coordination code on our side.\nThe ceiling is the partition count:\nmax_parallel_consumers = number_of_partitions\nIf a topic has N partitions, you can run up to N consumers in parallel and no more. So the partition count is the most consequential configuration we set on each topic. It's the basic knob that governs how far the SAYT event service can scale horizontally for that stream. We pick partition counts for the peak we can plausibly project, not the average. Throughput grows. Partitions are awkward to change after the fact.\nThe base requirement for the SAYT event service: be fast\nThe event service has \nmilliseconds\n, not seconds, to decide whether a given event matters. A database round-trip is a few milliseconds at p50 and tens at p99, which is already over budget on the rejection path. Doing one per event would crush the underlying store and push the consumer into permanent lag.\nSo the per-event filter is cache-resident, and the whole point of that cache is to avoid the database round-trip altogether on the hot path. The cache holds the SAYT plan configuration we need for filtering, keyed by the customer: which event types each plan covers, the amount or percentage to save, and the withdrawal preference. Every incoming event hits the cache first. The vast majority of events fail the filter at this stage and are dropped without ever touching a database. Only the small subset that passes the cache check proceeds to rule evaluation and topic publication.\n\nFigure 2: Event filtering funnel\nThe cache is invalidated on 'plan create, edit, or cancel', with the core service's database as the source of truth. We use a two-tier setup: a short-TTL local in-memory map per instance, backed by a shared Redis cluster, so a cold-starting instance doesn't stampede the database to rebuild its view. Hit ratio sits in the high nineties even under autoscaling churn, which means the database barely sees the firehose.\nWithout this cache, the architecture doesn't work. With it, the rejection path is essentially free, and the consumer keeps up with the bus regardless of how many customers are not transacting in a way SAYT cares about.\nEvent deduplication: let the database do the work\nAnyone who has worked with event buses long enough has stopped believing in exactly-once delivery. The honest design assumption is \nat least once\n. Events arrive twice, occasionally three times, especially when a consumer crashes mid-processing and the bus redelivers what it had already started working on.\nFor a service that moves money, "process the same event twice" translates directly to "credit the customer's SAYT pot twice for the same transaction, and pull the matching amount out of their main wallet twice." Not acceptable.\nThe naive approach is a two-step dedup: read the dedup store, and if you haven't seen the event, insert a record and proceed. That costs two database round-trips on every matched event and opens a race window between the read and the write where two consumers can both decide to proceed.\nWe do something simpler. Every event carries a unique event_id from its producer. We treat that column as a uniqueness constraint in the database itself, and we just \ninsert\n. If the event has been seen before, the database rejects the insert with a duplicate-key violation. We catch the violation, treat it as "already processed, drop", and move on.\n\nFigure 3: Insert-first deduplication using a unique constraint\nThis is faster on the happy path (one round-trip instead of two), correct under concurrency (the database serialises the constraint check, so two consumers racing on the same event_id can't both win), and operationally simpler because the invariant lives in the schema, not in application code.\nThe ordering matters. The unique-constraint insert is the side effect that determines whether the event has been processed, so it has to happen \nbefore\n publishing to the internal SAYT topic. If we published first, a crash between the publish and the insert would leave an instruction on the topic with no dedup record, and a redelivery would publish a second instruction. The customer's pot gets credited twice. The reverse ordering is recoverable: insert first, crash before publish, the saving silently doesn't happen, and we replay it. A much better failure mode than a phantom double credit that a customer cannot explain.\nThe same pattern applies on the core service side. The credit instruction itself carries a unique ID, and the actual credit uses that ID under the same insert-first pattern. The dedup boundary travels with the work as it crosses service lines.\nFor events that genuinely cannot be processed (malformed payloads, unknown customers, bad rule outputs), we use a dead-letter queue. After a small number of retries, the event is pushed to the DLQ with metadata describing why. The main consumer moves on. We monitor DLQ depth; in steady state, it's near-empty, and any growth is treated as an incident.\nA design choice: prioritise instant events, drop stale ones\nThis is the decision I'm proudest of, and the one most likely to surprise other engineers. SAYT is not a system that catches up. We deliberately drop events that arrive too late.\nEvery event carries a produced_at timestamp. The event service computes:\nevent_age = now() - event.produced_at\nif event_age > FRESHNESS_WINDOW:\n    log_stale_event(event)\n    return  # drop\nThe freshness window is short, on the order of minutes rather than hours. Long enough to absorb normal consumer lag and a brief outage. Short enough that a customer who looks at their app shortly after transacting still sees the savings land in real time.\n\nFigure 4: Freshness window decision\nThe reasoning is product-driven. SAYT is a real-time savings product. If a customer bought airtime two days ago, they've moved on. They've spent that money in their head. If SAYT suddenly moves money out of their wallet now for a savings rule tied to that purchase, the experience is alarming. They check their app, see a transaction they don't recognise, justified by a rule they may not even still want. We've harmed trust in trying to be thorough.\nSo we draw a line. Stale-event drop rate is monitored, and a sustained spike means something upstream is broken, but the act of dropping is not, in our system, a failure. It's the correct thing to do. The retries, the DLQ, and the durable log on the bus exist to maximise the chance of \ntimely\n processing and to make untimely outcomes visible. A small amount of completeness traded for a much larger amount of trust.\nThe architecture is a strategy\nThe reason this matters more than just "we built a savings product well" is that the same architecture is what lets SAYT become more than a savings product.\nThe event service does not know it's processing transactions. It's processing \nevents\n. The fact that today every event happens to be a financial transaction is a property of what we've wired up, not a property of the system. Adding a weather event source, a sports result feed, or a calendar trigger is wiring, not redesign. The same is true on the destination side. Today, the core service credits a SAYT pot. Tomorrow, that destination becomes a parameter: Target Savings, Flexible Savings, anything else we ship.\nThe pattern underneath is the difference between a service that \nresponds to commands\n and a service that \nresponds to facts\n. Direct integration is command-driven. A finishes its work and tells B what to do next. Event-driven design is fact-driven. A publishes a fact, and B decides whether to act on it. Most of the systems I've worked with that aged badly were command-driven, where they should have been fact-driven.\nIf you're building something that has to react to many other things, spend an extra week resisting the temptation to wire it up directly. Build the producer-bus-consumer shape. Split your consumer into a part for filtering and a part for business logic. You'll save years on every version after.\n____________________________________\nWe’re building systems that power millions of transactions every day. If that kind of engineering excites you, check out our \ncareers\n page.

Save as You Transact: How we built a savings product that reacts to millions of events

Most product teams are sitting on research gold they don't know how to use. They conduct interviews, gather feedback, and collect transcripts, then struggle to turn it all into something the team can actually act on. This is that story, and more importantly, this is the method that changed how I approach qualitative data.\n1. In the Beginning\nYou have carried out your user interviews and research, but what do you do with the messy data you have gathered? How do you make sense of it and use it to make informed decisions?\nI recently conducted user discovery for a new feature we shipped. I grouped users into three categories to understand them at every phase of the journey and to see how each step was perceived.\nI started by gathering users across the different groups and creating a research plan, which I shared with the customer research team for vetting. A process was then set up for interview slots, and the journey began. But in reality, this was only the beginning.\n2. Making Sense of the User Groups\nDividing users into groups wasn't just for structure. It was the only way to get a complete picture of the journey.\nEach group represented a different stage. Group one had completed the process. How did you complete it, and what were the hurdles you had to cross, if any? Group two was still in the process. Are you experiencing similar hurdles or a different one entirely? Group three hadn't started the process. What prevents you from starting this flow?\nUnderstanding every stage helps uncover pain points and how to make the flow seamless for users throughout the cycle. A single user group would have collapsed very different experiences into one. We would have missed that the blockers at the start of the journey are completely different from those at the end.\n3. Carrying Out the User Interviews\nBefore the first call, make sure you have recording and transcription enabled on your meeting platform. Get approval from the user upfront. This is non-negotiable.\nA plan? Check. Interviews scheduled? Check. Now, this is where you start acquiring data. A bunch of user interviews are lined up for the week; the interesting part is asking questions and listening to your users talk. Because you have a plan set up with your question bank, it makes it easier, but a user may say something you are not expecting, and it is important to ask follow-up questions to dig deeper.\nWe have AI infused into our meeting platform, so recording and transcribing were seamless. Tools like Otter.ai, Fireflies, or even the built-in transcription on Google Meet and Zoom work well here. The transcript becomes your best friend in the next phase. Trust me on that.\n4. The Storm Before the Calm\nPhew. The interviews are done, and it has been quite a week. Twenty calls across three different user groups, and now I am sitting with a folder full of transcripts and recordings staring back at me.\nHere is the thing nobody really tells you about user research. The interviews are actually the fun part. You are in a flow state, asking questions, listening, picking up insights in real time. Post-interview is when the actual work begins. You have raw data and need to analyse it and draw insights.\nI opened the first transcript. Then the second. By the fifth one, I already had contradicting information. A user from group one said the process was straightforward. A user from group two said they had no idea what they were supposed to do at the exact same step. Both were right. They were just at different points in their journey. But reading transcript by transcript, it is genuinely hard to hold it all together.\nI needed a way to find patterns from these conversations without losing the nuance. That is when I turned to thematic analysis.\n5. Introducing Thematic Analysis\nThematic analysis is a way to form patterns from qualitative data. You break down your data into labels called codes, and then group those codes into bigger themes.\nHere is how I approached it. I gathered all the transcripts and read through them, coding and labelling each one. A code is simply a short label that captures what a user is saying or feeling at that point. For example, a user said, "It took me 5 minutes to complete," and the code for that was completion time. Another said, "One thing I like is that delivery is swift", and the code was ‘merchant satisfaction’. Another said, "I don't think anyone should have issues filling the information", and the code was ‘perceived difficulty’.\nDoing this across 20 transcripts, you start to pick up patterns. The same ideas are showing up just in different words. That is your signal.\nNext, group similar codes into themes. Codes like 'completion time', 'perceived difficulty', and 'document readiness' all pointed to the same underlying problem. They became one theme: Activation Friction. \nWhere you have a cluster of related but distinct codes under one theme, those become sub-themes, a way of preserving the detail without losing the bigger picture.\nBy the end, I had 8 themes that cut across all three user groups. What made this powerful was seeing which themes were universal across the journey and which ones were specific to a particular stage.\n6. Where AI Fits In\nNow let's address what everyone is thinking. Reading through twenty transcripts? Three groups? Coding responses manually? This sounds like a lot, and this is where AI can come in to make your work significantly easier.\nOnce I had my transcripts ready, I fed them into an AI tool. ChatGPT works well for this, as does Claude or a dedicated research tool like Dovetail. I prompted it to code responses, identify similar phrases, and flag recurring sentiments. What would have taken me an entire day was done in a fraction of the time.\nHere is something important, though. AI gives you a strong starting point, not a final answer. The interpretation of what those patterns actually mean for your product still sits with you. You handle the thinking.\n7. From Themes to Product Insights\nHaving 8 themes feels good. But themes alone don't ship features. The next step was turning what I found into something the team could actually act on.\nI broke down each theme into four sections in my research report.\nWhat We Heard. This section was a brief summary of what users said about the theme, with sample quotes to support it. For example, under the theme of Activation Friction, quotes clustered around not knowing a document was required, or needing time to gather specific documents before completing the flow.\nWhy It Matters. How does this theme affect the user journey? Does it cause delayed activation? Drop-off? Frustration that only surfaces later? This section connects the theme to a real business or experience consequence.\nWhat It Means. This is where you translate the theme into a product diagnosis. Where does the issue actually come from? Is the form too long, or is it the missing visual cues that tell users what to prepare? Are users benchmarking against a competitor's experience? This section separates the symptom from the cause.\nRecommendations. Based on everything identified, what should the team do? This is where research becomes a roadmap. Before the analysis, conversations with stakeholders were centred on surface-level fixes. After that, we were talking about the right problems.\n8. What Thematic Analysis Changed\nHonestly? It changed how I think about qualitative data altogether.\nThematic analysis gave me a process, and having a process meant I could defend my insights. Walking into stakeholder meetings, the conversation wasn't "users seem frustrated with the journey." It was saying "across all three user groups, activation friction and perceived difficulty were the two most consistent themes, and here is the\nIf you are a product manager sitting on a pile of interview transcripts right now and wondering where to start, start here. The themes are already in your data. Thematic analysis just helps you find them. The messy data was never the problem. It was always the signal.

Turning Messy Data into User Insights: Using Thematic Analysis

Research Presented by Prof Adeolu Afolabi at Moniepoint Research Talks, March 2026\nMoniepoint Research Talks is our monthly engineering chat, where research meets practice. Each month, we host academics and senior engineering experts to share insights that help engineers build better systems, solve real-world problems, and advance their craft.\nIntroduction\nThe world of financial technology changes fast. In 2026, banks and payment apps use smart tools like AI and blockchain to keep money safe. Companies in Nigeria like Moniepoint, Kuda, and Flutterwave lead the way. They help people pay bills and get loans through mobile phones. Since everything is digital, these platforms must be very strong. If a digital platform has even one small hole, thieves can cause big trouble. Security is the most important part of any banking app today.\nWho Uses Fintech?\nTo understand security, we must look at who uses these tools. There are four main groups in the fintech world. Each group needs protection from hackers.\nBanks to Banks:\n This is where large banks use technology to communicate with other banks. It is called business-to-business or B2B.\nClients to Banks:\n This refers to a large company or client working with a bank.\nBusiness-to-Consumer:\n This is when small businesses provide services to consumers.\nConsumer-to-Business:\n This is the most common way regular people use apps to pay for goods and services.\nEach group has different needs. A big bank needs to move millions of dollars safely. A small shop owner just needs to make sure their daily sales do not go missing. Because there are so many users, the risks are always growing.\nModern Cyber Threats in 2026\nThieves have become very smart by 2026. They no longer just guess passwords. They use high-tech tools to trick people. Here are the top threats facing users and banks today.\nAI-Powered Phishing\nPhishing is when a thief sends a fake message to steal your info. In 2026, they use AI to make these messages look perfect. They send texts on WhatsApp or SMS that look exactly like your bank’s official mail. Many people fall for these because they look so real. If a person does not know much about digital tools, they are at high risk.\nDeepfake Voice Fraud\nThieves now use AI to copy voices. They can make a phone call sounding like a bank boss or a family member. They use this fake voice to ask for money or a password. This is why authentication is so important. Banks must check if the person on the phone is actually who they say they are. These attacks happen very fast, making them hard to stop.\nSleeper Account Fraud\nSometimes, people open a bank account and do not use it for months. These are called dormant or sleeper accounts. Criminals look for these quiet accounts. They use them to move stolen money around. It is a big blind spot for many computer systems.\nSIM Swapping and OTP Theft\nYour phone number is a key to your bank. In a SIM swap, a thief takes over your phone number. They then receive your One-Time Password (OTP) on their phone. With the OTP, they can reset your banking password and steal your money. This happens when phone companies have weak security.\nKYC Farming\nKYC stands for "Know Your Customer." Banks use ID cards to verify users. Thieves now use real ID cards to create many fake accounts. These are called mole accounts. They use these accounts to hide their tracks when they commit crimes.\nFinancial Losses and Challenges\nCybercrime costs a lot of money. In Nigeria, the numbers show how much is at stake. Even though we are getting better at stopping it, the attempts are still very high.\nYear\nMoney Lost to Fraud (Naira)\nStatus\n2024\n52.26 Billion\nVery High\n2025\n25.85 Billion\nDecreasing\nThe amount of money lost went down in 2025, but it is a lot. When people lose money, they stop trusting digital banks. This makes it harder for the country to grow its economy.\nRegulatory Problems\nThe Central Bank of Nigeria (CBN) has new rules to stop these crimes. However, there are still many challenges. Sometimes it takes too long to get approval for new security tools. About 62.5% of the time, there are delays in getting site approvals. This slow pace helps the criminals stay ahead.\nCommon Financial Crimes to Watch\nWe must look at how criminals move money. They use several tricks to bypass the rules.\nOnline Money Laundering:\n Criminals use cryptocurrency to hide where their money came from. These funds are often hard for the bank to track.\nFake Loan Creation:\n Thieves use "synthetic identities." This is when they mix real and fake info to create a new person. They use this fake person to take out loans they never pay back.\nPayment Fraud and Chargebacks:\n Some users lie and say they never bought an item. They get their money back, but the business loses out.\nPOS Malware:\n Software that attacks point-of-sale (POS) machines. It steals data when you swipe your card at a store.\nThe Layered Defence Plan\nTo stop these crimes, we need a layered plan. If one layer fails, the next one stops the thief. There are four main layers of security.\nPhysical Security: \nThis protects the actual hardware and buildings. It uses tools like fingerprint scanners to let people in.\nNetwork Security:\n This uses firewalls to block bad traffic. It keeps the internet's "pipes" safe.\nApplication Security:\n This protects the app or software itself. It makes sure the code has no holes.\nData Security:\n This involves encrypting the data. Even if a thief steals the data, they cannot read it without a special key.\nUsing all these layers makes the system hard to break. It creates a safety net for every transaction.\nUsing AI for Better Detection\nOld computer systems use simple rules. For example, a rule might be "stop any payment over $5,000." But thieves know these rules. In 2026, we use Machine Learning (ML) to spot weird behaviour.\nSupervised Learning\nWe train the computer by showing it what fraud looks like. It learns the difference between a good buyer and a thief. We use tools like Decision Trees and Support Vector Machines to label transactions.\nUnsupervised Learning\nThis is even smarter. The computer looks for patterns on its own. It finds "outliers" or strange actions that do not fit the normal routine. It can find new types of fraud that we have never seen before.\nGraph Neural Networks\nThis tool looks at how different accounts are connected. If ten different accounts are all sending money to one person in a strange way, the computer flags it. This helps stop big crime networks.\nBehavioural Biometrics: The Future of Security\nPasswords are not enough anymore. In 2026, we look at who you are and how you act. This is called behavioural biometrics.\nThe system can track:\nHow fast you type on your phone.\nHow much pressure you use on the touch screen.\nThe way you move your computer mouse.\nThe way you hold your phone or walk.\nIf a thief steals your phone and tries to use your app, the system will notice. They do not type like you. They do not hold the phone like you. The app will lock itself instantly. This is a very strong way to stop unauthorised users.\nBetter Security Strategies\nBanks are moving away from old habits. They now use a "Zero Trust" model. This means the system trusts no one. Every single time someone tries to move money, the system checks their ID. It does not matter if they just logged in a minute ago.\nBanks also use Red Teaming. This is when a team of good hackers tries to break into the bank. They find the weak spots before the bad guys do. This helps the bank fix problems early. They also scan their systems every day to find any new bugs.\nThe Human Side of Security\nTechnology is only half of the battle. People are the other half. Even the best AI cannot save a bank if an employee helps a thief. This is called an insider threat.\nTo stay safe, companies must:\nTrain Employees:\n Every worker must know how to spot a fake email.\nReport Early:\n If something looks wrong, workers must say something right away.\nBuild Ethics:\n Workers need to have a strong sense of right and wrong.\nWork Together: \nThe tech and office teams must communicate with each other.\nWhen everyone works together, the whole system becomes much safer.\nConclusion\nCybersecurity in 2026 is a constant battle. As thieves use AI to steal, banks must use AI to protect. A good security plan must have many layers. It needs to use physical locks, smart code, and careful data tracking. By using tools like Zero Trust and behavioural biometrics, we can stop fraud before it starts.\nThe goal is to create a safe place for everyone to use money. Nigeria's fintech world is growing fast. If we keep our systems strong and our people trained, we can build a future where everyone's money is safe. We must stay ready for new threats and keep improving our tools every day.\n
\nAbout the Speaker\nProfessor Adeolu O. Afolabi is a Professor of Information Security and E-Learning in the Department of Cyber Security at Ladoke Akintola University of Technology (LAUTECH).\nHe holds a PhD in Computer Science from Obafemi Awolowo University (OAU) and has built a distinguished career at the intersection of cybersecurity, research, and academia. With over 60 peer-reviewed publications and four textbooks to his name, he is a leading voice in information security research.\nHe is a member of several professional bodies, including the Computer Professionals Registration Council of Nigeria (MCPN), the Nigeria Computer Society (MNCS), and the International Association of Engineers (IAENG), and currently serves on the Governing Council of LAUTECH.\n

How to Protect Fintech Systems from Cyber Attacks in 2026

The Performance Conversation That Changed the Room\nDuring a performance review, an engineering manager highlighted that an engineer had spent several late nights debugging production issues and pushing features. It was presented as proof of performance beyond expectation.
The room expected agreement.
Instead, I said something that made it uncomfortable.\nSpending more time is not the same as performing better. Effort alone is not performance. Outcome is.\nYes, features were shipped quickly. But months later, we were still fixing subtle bugs tied to that release. Every small enhancement required touching of core architectural decisions. During the product review, assumptions were not questioned. The design was implemented verbatim.
\nThe engineer looked busy. The system was fragile.\nIn the same period, another engineer worked normal hours. No heroics. No drama. Since his release, there have been no production incidents in that module. He questioned the PRD. He challenged design decisions. He wanted to understand how the feature affected customers. As we scaled, extending his work became easier, not harder.
He was rated beyond expectations.
That contrast exposed something deeper — a flaw not in the engineers, but in what we had trained ourselves to value.\nWe Reward What Is Visible, Not What Is Valuable\nMost engineering performance systems are biased toward visible work.
Hours spent. Story points completed. Tickets closed. Production calls attended.
These are observable. They are measurable. They make managers feel progress is happening.
But systems thinking is often invisible.
Clean architecture is invisible. Prevented incidents are invisible. Reduced complexity is invisible. A system that rarely breaks does not receive applause.
Over time, engineers optimise for busyness instead of durability.
When that happens, ticket completion becomes the goal instead of system integrity.
And nowhere is this more apparent than in how we treat incidents.\nPrevention Should Matter More Than Recovery\nRecovery is important. Incident response discipline matters. Mean time to recovery matters.
But prevention should carry more weight.
If I were to assign weights, prevention should count for 80 per cent and recovery for 20 per cent. It is cheaper to prevent than to recover.\nReducing incidents from 10 per month to 1 is more valuable than reducing recovery time from 2 hours to 30 minutes.
When teams focus primarily on recovery, they build better firefighters.\nWhen teams focus on prevention, they build better architects.
If you celebrate the firefighter more than the architect, you are building a culture that creates a fire incident all the time.
The root cause of this pattern is almost always the same — teams that have been conditioned to think in tickets.\nTicket Thinking Creates Avoidable Work\nTicket thinking creates the illusion of velocity while increasing fragility.\nArchitecture gets reworked after release. Hotfixes stabilise features that should have been designed more carefully. Refactoring becomes necessary because shortcuts were taken to meet sprint targets. Adding new modules becomes increasingly expensive because the foundation is unstable.\nVelocity looks impressive in the short term. But complexity compounds quietly.
Every new feature becomes harder to ship safely. Every release carries anxiety. Engineers spend more time fixing than building.
That is not scale. That is activity disguised as progress.
So how do you break the cycle?\nHow to Build Teams That Think in Systems\nShifting from ticket thinking to system thinking requires deliberate leadership.\nRedefine performance: \nStop rewarding just hours spent or story points completed. Start recognising engineers who reduce incident frequency, simplify architecture, and make future changes easier. Promotion should reflect ownership of outcomes, not volume of output.\nChange what you measure:\n Track incident recurrence, change failure rate, and the safety of extending the system. Measure the reduction of avoidable work. If every new feature destabilises the system, velocity is meaningless.\nInvolve engineers earlier:\n Engineers should not simply implement requirements. They should challenge assumptions, clarify ambiguities, and consider second-order effects. An engineer who takes a PRD verbatim is implementing tasks. An engineer who questions it owns outcomes.\nReward complexity reduction:\n Promote engineers who simplify flows, clarify domain boundaries, and reduce cognitive load. Complexity is invisible until it explodes. Recognise those who prevent that explosion.\nWhen you start doing all of this, one question inevitably comes up — especially from engineers who have been grinding hard but not seeing the results they expect.\nPromotion Is About Mastery, Not Busyness\nMany engineers ask why they are not being promoted despite long hours and visible effort. Promotion panels are not looking for busyness. They are looking for depth.\nCompanies are looking to promote masters.
Mastery in engineering is not about writing more code or responding to more incidents. It is about understanding systems deeply enough to design them so that problems do not repeatedly occur. It is about building scalable foundations and optimising for the long term.
Speed and quality are not enemies; they answer to mastery.\nThis is ultimately a leadership problem. If we want to build engineering teams that think in systems, not tickets, we must reshape how we define performance, what we measure, and what we celebrate.\nTickets close. Systems endure.\nAnd endurance is what separates coding from engineering.\nAbout The Authour\nJohn Ojetunde leads the Savings, Account Payments and POS Channels Engineering Teams at Moniepoint. 

How to Build Engineering Teams That Think in Systems, Not Tickets

\nOver a month ago, I sat on a panel at\n AI Everything MEA in Cairo\n, discussing trust, transparency, and accountability in AI with two investors and one of the sharpest tech journalists in the business. I was the only operator on stage, the person who builds the systems rather than evaluating them for their investment opportunities and viability.\nI want to discuss what AI trust actually looks like from the inside.\nThe Question That Changed the Conversation\nThe moderator, Mike Butcher, asked a version of a question I hear constantly: What should investors look for when evaluating AI companies? The panel of investors -Yehia Houry from Flat6Labs and Abdelrahman Hassan from Enza Capital-offered thoughtful insights on governance frameworks and founder credibility.\nThen it was my turn. And I decided to start with a story about SMS charges and their impact on computing Monthly Active Users (MAU).\nAt a financial institution like ours, customers earn interest each month under conditions set by the Central Bank. At the start of each month, the system debits SMS notification charges for the period and posts interest income for that same period (if conditions are met). These are system-generated transactions — the customer did not do anything. They did not open the app. They did not transfer money. They did not make a purchase.\nBut if your definition of “monthly active user” is “any customer with at least one transaction,” those customers show up as active. Here’s what happens. Marketing reports a high Monthly Active User (MAU) to the board. Product looks at the same data and sees low engagement — average transaction values are depressed because genuinely active customers are mixed with essentially dormant ones. Finance sees a third picture entirely because they’re tracking revenue-generating activity, and neither transaction is revenue-generating.\nThree teams. Same underlying data. Three different stories.\n\nVaried MAU definitions (Image generated by Claude)\nAnd here is the part that matters for AI: if you build churn prediction, credit scoring, or personalisation on top of this data, the AI inherits the confusion. It treats a dormant customer with an SMS debit as if they were transacting daily. Every downstream model is making decisions based on a definition nobody agreed on. That’s not a model problem. The model is doing exactly what you told it to. It’s a governance problem. And it starts with something as simple as: what does “active” actually mean?\nI could feel the room shift when I told that story. Not because it was dramatic, but because every founder in the audience had either experienced it or was currently living with it.\nHow Governance Got Built Into Our Architecture\n\nFull chain of custody (Image generated by Claude)\nAt Moniepoint, we process over 100 billion transactions across multiple entities in Nigeria and the UK. Our reconciliation systems, our analytics platforms, our financial reporting, all of it depends on data being trustworthy. But we did not start with a governance strategy. We started with a constraint.\nIn Nigeria, most banks don’t have APIs capable of handling the transaction volume we process (I know this is a bit of a humblebrag, but this is our reality). So bank statements had to be uploaded manually into our reconciliation system. Immediately, that creates a trust problem: if the same person who uploads a statement also approves it, you have no verification. So we built a maker-checker system. The uploader cannot be the approver.\nThat was not a compliance decision. It was an operational necessity. But it established a principle that carried through everything we built afterwards. Settlement reports, which translate the bank’s view of a transaction into a form comparable to our internal records, have the same level of traceability. When our system auto-reconciles a transaction, it records not only the match but also which pipeline made the decision and who built that pipeline. If a data engineer created the tagging logic and that logic categorised a transaction a certain way, you can trace it back. This outcome exists because of this pipeline, built by this person, using this logic.\nEvery transaction at Moniepoint has a full chain of custody. From the moment someone pays at a terminal, through settlement, through reconciliation, and onward into our ERP for financial reporting and analytics. I think of it as the life of a transaction.\n Like a note in a RioPY composition — from the moment it is struck to the second it resolves, every vibration is intentional. Remove one stage, and the harmony collapses into noise\n.\nWe did not build this because a regulator told us to. We built it because Nigerian banking infrastructure gave us no choice. And now that the world is calling for AI governance, traceability, accountability, and human oversight, we’re well on our way. Not (just) because we were visionary, but because the constraints forced good architecture.\nThe SQL Problem Nobody Talks About\nThere’s a moment in every growing organisation when data becomes central to how people do their jobs. At Moniepoint, that moment came as our finance team expanded. Everyone needed answers from data. The default solution was obvious: learn SQL. Write your own queries. Get your own numbers. And it worked. For a while. Here's what really unfolded. Finance professionals, those experts brought in to analyse trends, shape strategies, and make key decisions, were dedicating hours to writing SQL queries, troubleshooting table joins, and determining which of the three tables contained the correct revenue data. They became tactical instead of strategic. And worse: ten analysts writing their own queries against the same data, with slightly different logic, were getting slightly different answers. The same fragmentation problem as the MAU example, multiplied across the entire team.\nThat was when we realised the answer was not “make everyone learn SQL.” The answer was to build a layer where the data is already governed, already defined, and accessible without writing code. That’s what drove us to build a conversational analytics interface where a finance team member can ask “what is this month’s revenue by entity?” in plain English and get a trustworthy answer. But — and this is the critical point I made on stage — our conversational analytics interface only works because the governance layer exists underneath. The canonical definitions. The data lineage. The shared vocabulary. Without governance, a conversational AI interface is just a chatbot writing bad SQL faster. The governance isn’t a feature of the AI. Governance is what makes AI possible.\nWhat AI Is Actually Doing (And Why It Needs Governance)\n\nDifferent Layers to AI Data Governance (Image generated by Claude)\nOne thing I noticed during the panel: when people talk about “AI in fintech,” they tend to imagine chatbots or credit scoring models. The reality, at least in our world, is more infrastructural and less glamorous.\nOur system automatically reconciles millions of daily transactions, matching internal records with bank data. It identifies, categorises, and often resolves exceptions without human help. Every automated match is fully auditable: we record the decision-making pipeline, logic, and data. This allows us to trace any incorrect match back to the failing rule or specific data, unlike relying on opaque AI outputs.\nTraceability allows us to trust the automation. Without it, we’d need humans reviewing every transaction, which is impossible at scale. With it, humans review exceptions — the cases the AI flagged as uncertain — while trusting that the automated decisions are traceable and reversible. And the same principle extends downstream. Our revenue and expense assurance frameworks continuously validate whether transactions are correctly classified and whether amounts flow correctly across entities. Our conversational interface translates natural-language questions into structured queries. Each layer of AI capability sits on top of the same governance foundation. Remove the traceability, and you have a black box processing billions of naira. Add it, and you have an auditable, explainable system that a regulator, an investor, or an auditor can interrogate.\nThe Honest Journey\nThe moment on the panel that felt most vulnerable was when I was asked whether founders should present a polished or an honest governance narrative.\nI chose honest.\nI told the room that Moniepoint’s AI capabilities began in different teams, with my team, Finance Systems, among the early adopters. We identified operational problems that could not be solved manually at our scale and built solutions. Reconciliation automation. Conversational analytics. Revenue assurance. These work. They are in production. They’re processing real money.\nBut they were built for one domain. The definitions, the data pipelines, the governance — they served finance. When we looked across the organisation, we saw other teams building their own versions, with their own definitions and assumptions. The AI worked in each silo. It didn’t compose across the company. So now we’re doing the harder work: building an enterprise data framework that centralises definitions, governance, and data standards across the organisation. It is not a finished story yet. It’s an active project.\nI said on stage: “Most companies pitch Phase 3 while living in Phase 1. They sell a vision of total AI governance while their internal teams are still arguing over which spreadsheet is the ‘source of truth.’ At Moniepoint, we are firmly in Phase 2: building the foundation. It’s unglamorous, it’s difficult, and it’s the only reason our AI is actually worth the compute it runs on.” The investors on the panel validated this immediately. Both said they would rather hear an honest account of where a company is and where it’s going than a polished narrative that falls apart in due diligence. One of them made a point I’ll remember: a company that can articulate its gaps has a plan to close them.\nThe “Operator’s Audit”: 3 Questions Every AI Investor Should Ask to Understand Data and AI Governance\n\nOperator’s Audit (Image created by Gemini)\nIf you are evaluating an AI company — or any company claiming “AI-driven” growth — ignore the pitch deck. Ask these three questions instead. They will tell you more about the company’s viability than any accuracy benchmark.\nThe “Triangulation” Test\nThe Ask:\n \n“Show me your Revenue (or MAU) metric across three different dashboards: Finance, Product, and the Board Report.”\nThe Goal:\n If the numbers match, they have \nData Governance\n.\nThe Red Flag:\n If the numbers differ and the team spends ten minutes explaining “why the logic is different for Marketing,” they don’t have an AI problem — they have a \nTruth Problem\n. You cannot build reliable AI on top of a fragmented truth.\nThe “Semantic” Test\nThe Ask:\n \n“Who owns the definition of ‘Active User’ or ‘Revenue’?”\nThe Goal:\n You’re looking for a single source of accountability — a “Data Steward” or a centralised governance layer.\nThe Red Flag:\n If the answer is “It depends” or “We all kind of just know,” the company doesn’t have data governance; it has \nData Opinions\n. AI trained on opinions is just a high-speed hallucination engine.\nThe “Post-Mortem” Test\nThe Ask:\n \n“Tell me about the last time your data was wrong. How did you catch it, and what was the ‘Chain of Custody’ for the fix?”\nThe Goal:\n Genuine governance requires \nTraceability\n. A mature company can point to the specific pipeline, the specific logic, and the specific person who corrected it.\nThe Red Flag:\n A company that says “our data is never wrong” either hasn’t deployed at scale or isn’t looking. If they can’t trace an error, they can’t trust an insight.\nGovernance is the Engine, not the brake.\n\nGovernance is the engine, not the brake (Image generated by Gemini)\nIf there’s one idea I want people to take from that panel (and from this article), it’s this: governance is not overhead. It’s not a compliance cost. It’s not a box you tick for regulators.\nGovernance is the engineering discipline that makes everything else possible. AI capabilities, investor trust, regulatory compliance, operational scale — all of it sits on top of whether your data is governed, your decisions are traceable, and your definitions are shared.\nThe companies that build this into their architecture from day one don’t just avoid regulatory risk. They move faster than everyone else when it’s time to deploy AI. Their teams ask questions and get consistent answers. Their models train on data that means what it says. Their audits are straightforward because the traceability already exists. Their competitors who skipped the governance step? They’re retrofitting. And retrofitting governance onto systems that were never designed for it is expensive, slow, and fragile. We learned this the hard way, through the constraints of Nigerian banking infrastructure and the operational demands of processing over 100 billion transactions. The constraints forced good architecture. The architecture enabled AI. And the AI, built on governed data, is trustworthy — not because we say so, but because anyone can trace any decision back to its source.\nThat’s what trust looks like. Not a pitch deck. Not a policy document. A production system where every transaction has a life story, and you can read it from beginning to end.\nAbout the Author\nWole Olorunleke is the Vice President, Finance Systems at Moniepoint, where his team builds the systems that process and reconcile over 100 billion transactions across Nigeria and the UK. He spoke on the “Trust, Transparency & Accountability: What Investors Want from AI Founders” panel at AI Everything MEA Egypt 2026 in Cairo, alongside Yehia Houry (Flat6Labs), Abdelrahman Hassan (Enza Capital), and moderator Mike Butcher (TechCrunch).

What I Learned About AI Trust from Reconciling over 100 Billion Transactions

Many developers face the same nightmare. You launch a service, and suddenly the database is overloaded. This leads to a total system collapse. You might ask yourself how many threads you need or how large your connection pool should be. Most people try to fix this by guessing. They run a load test or scale up slowly and hope for the best.\nThere is a better way to solve these problems. You can use math to predict how your system will behave. This is called Queueing Theory. Everything in computing is a queue. The way your operating system schedules tasks and the way network packets move are all queues. If you understand the math behind these queues, you can build systems that do not crash under pressure.\nWhat Is a Queue?\nA queue is just a line. You see them at the bank or the hospital. In a computer, a queue holds requests waiting for a server. There are three main parts to any queue:\nArrival Rate: This is how fast new requests come in.\nService Time: This is how long it takes the server to finish one request.\nService Rate: This is the number of requests the server can handle per unit time.\nIf you do not plan your queue well, the line will grow too long. Long lines mean slow response times for your customers. If you add too many servers, you waste money and resources. Queueing theory helps you find the perfect balance.\nThe Problem with Independence\nMost math models assume that every request is independent. This means that one person joining the line doesn't affect the next person. In the real world, this isn't always true. Think about a thundering herd. If a service fails, everyone tries to reconnect at the exact same time. The arrival rate spikes because the previous requests failed. Your math needs to account for these bursts.\nKey Parts of a Queue System\nTo model your system, you need to understand its parts. Engineers use something called Kendall's Notation to describe these systems. You might see terms like MM1 or MMC. These are just codes for how things arrive and how many servers you have.\nQueue Discipline: This is how you pick the next item to process. Most systems use "First In, First Out" (FIFO). Sometimes it is better to skip old requests and process new ones first to keep the system moving.\nSystem Capacity: The limit on how many requests can wait. Many businesses want infinite capacity, but that can lead to hidden overloads.\nPopulation Size: The total number of potential customers. For a doctor, it is the number of sick people in town. In a web app, we usually treat this as infinite.\nThe Rule of Utilisation\nThe most important number in queueing theory is utilisation. You calculate this by dividing your arrival rate by your service rate. It tells you how busy your server is.\nYou must keep your utilisation below 100%. If your utilisation hits 1.0, your queue will grow forever. In fact, you should aim for much lower. Once you pass 70% utilization, your system will likely start to struggle.\nThe Hockey Stick Effect\nWhen utilisation stays low, the wait time is short. But as you get closer to 100%, the wait time does not just grow a little bit. It explodes. This is called the hockey stick effect. If your CPU or database is at 90% utilization, a tiny spike in traffic will cause a massive backup. It becomes very hard for the system to recover from this state.\nLittle's Law: The Simple Formula\nYou do not need to be a math genius to use these ideas. A man named John Little created a simple rule called Little's Law. It is an approximation that works for almost any system.\nThe formula helps you find the average number of items in your system. If you know how fast requests arrive and how long they take to process, you can find your queue length.\nAverage Number of Items = Arrival Rate x Average Wait Time\nThis formula is very helpful for tuning your settings. You can use it to figure out how many threads your Tomcat server needs. You can also use it to set the size of your network queue. It hides the complex math and gives you a clear target.\nWhy Variance Matters\nEven if your average speed is good, variance can kill your performance. Variance means that some requests are much slower than others. If one request takes a long time, it blocks everything behind it.\nThis is why you should look beyond the average. You should track these metrics:\nP50 (Median): The middle speed.\nP99: The speed for the slowest 1% of requests.\nYou want your P99 to be as close to your median as possible. If there is a big gap, your system has high variance. High variance makes it much harder to predict when your system will crash.\nPractical Steps for Engineering\nYou can apply these rules to many parts of your tech stack. Here are three common areas where queueing theory helps:\n1. Connection Pools\nDon't just guess your database connection pool size. Treat each connection as a server. If your queries take 10 milliseconds and you get 90 queries per second, you can calculate the exact pool size you need. Aim to keep the pool utilisation under 70%.\n2. Thread Pools\nYour application has a limit on the number of threads it can run. If the thread pool is too small, requests wait too long. If it is too large, the CPU wastes time switching between them. Use Little's Law to find the right number of threads based on your arrival rate.\n3. Fail Fast with Timeouts\nDon't let people wait forever in a queue. This is called an unbounded queue. It hides the fact that your system is overloaded. Instead, set a short timeout. If the queue is full, tell the user to try again later. This is called failing fast. It prevents requests from piling up and crashing your entire network.\nComponent\nWhat it Represents\nGoal\nArrival Rate\nNew user requests\nMeasure and monitor\nService Rate\nDB or CPU speed\nImprove with better code\nUtilization\nSystem load\nKeep under 70%\nK (Capacity)\nQueue limit\nLimit to avoid hidden lag\nConclusion\nQueueing theory is a powerful tool for any developer. It moves you away from guessing and toward making smart decisions. Most production issues arise from a database overload or an incorrect connection pool. By understanding arrival rates and utilisation, you can stop these problems before they start.\nRemember to keep your systems stable. Do not let your utilisation climb too high. Watch your variance and try to keep your slowest requests under control. If you use these mathematical models, you can build services that stay fast and reliable even when traffic spikes. Applying just a little bit of science to your scaling will save you hours of debugging in the future.\n
Ope communicates this in detail in \nthis video\n

How to Use Queueing Theory to Stop Your Database from Crashing

Introduction\nMoniepoint Research Talks is our monthly engineering chat, where research meets practice. Each month, we host academics and senior engineering experts to share insights that help engineers build better systems, solve real-world problems, and advance their craft.\nIn this edition, Professor Babajide Samuel Afolabi from Obafemi Awolowo University reminded us of a time when sending money to a student meant handing cash to a bus driver and waiting 21 days for it to arrive or paying exorbitant fees for postal orders. While fintech has revolutionised speed and access, a significant gap remains: trust and true inclusion.\nAt Moniepoint, we pride ourselves on building reliable infrastructure for businesses. But as Professor Afolabi highlighted in his talk, "User Modelling for Enhanced Fintech Services," technology alone isn't enough. To truly serve the 25% of Nigerian adults who remain unbanked, we must move beyond standard demographics and build digital "mirrors" that reflect real human behaviour. This post summarises the key takeaways from his session, exploring how engineers can use data to bridge the gap between code and customer reality.\nKey Concepts Covered\nUser Modelling:\n Creating digital representations of users based on data and behaviour.\nThe Trust Gap:\n The difference between urban risk-taking and rural reliability.\nPersona Development:\n Moving beyond "users" to specific profiles like "Amina" or "Chinedu".\nBehavioural Forensics:\n Using metadata (like MAC addresses) to detect fraud.\nWhat is User Modelling?\nProfessor Afolabi defines user modelling as "building a mirror that helps us to see our users more clearly". It is not just about logging transactions; it is about creating a dynamic profile that helps in personalisation, prediction, and segmentation.\nFor engineers, this means shifting focus from what a user does (clicks, transfers) to who the user is.\nUrban Users:\n Prioritise speed and instant delivery. They are risk-takers but lack loyalty (trust is not in their vocabulary).\nRural Users:\n Prioritise safety and reliability. They are sceptical, but once you earn their trust, they remain loyal indefinitely.\nWhy It Matters: The "Egg" Analogy\nProfessor Afolabi shared a powerful story about financial literacy. A woman was given capital to start a small business selling bread and eggs. On her first day, she became hungry, boiled six eggs, and ate a loaf of bread, consuming her capital before making a sale.\nThe Engineering Takeaway: We cannot simply "throw technology" at this problem. If a user doesn't understand that capital isn't revenue, a complex dashboard won't help them. We must design interfaces that guide behaviour, perhaps using voice prompts for the illiterate or simplified "safe-to-spend" balances.\n3 Steps to Implementing User Modelling\nBased on the talk, here is a framework for implementing better user models in our stack.\nStep 1: Data Audit and Segmentation\nWe must look beyond standard transaction logs. Useful data sources include:\nTransaction frequency:\n Does the user withdraw immediately after receiving funds?\nDevice usage:\n Are they on a smartphone or a feature phone?\nExternal data:\n Regulatory data and customer support interactions.\nAction:\n Audit your current data pipelines. Are you capturing the "silence" from users who aren't providing feedback?\nStep 2: Develop Detailed Personas\nDon't design for "everyone." Design for specific archetypes. Professor Afolabi outlined three key personas relevant to our market:\nAmina (The Rural Trader):\n Uses POS for daily sales, may have low literacy, needs voice prompts.\nChinedu (The Urban Student):\n Tech-savvy, seeks microloans, price-sensitive.\nNgozi (The MSME Owner):\n Needs payroll management and staff oversight tools.\nStep 3: Predictive Modelling for Fraud\nBehavioural modelling is our best defence against fraud.\nThe Case Study:\n During a university exam, tracking shifted from IP addresses to MAC addresses. The team discovered that 100 students had submitted exams from the same device within 7 hours, exposing an AI-cheating ring.\nApplication:\n In fintech, we can flag accounts when behavioural patterns (such as device usage or location) suddenly shift, helping catch fraud before it occurs.\nA Mirror on Moniepoint: Strengths and Weaknesses\nOne of the most valuable parts of the session was Professor Afolabi’s candid assessment of Moniepoint’s current market standing.\nWhat We Are Doing Right\nMSME Focus:\n We are correctly identified as the champion of the informal sector, including bread sellers and grocery owners.\nReliability:\n Our POS terminals are often called "personal ATMs" because they continue to work when traditional bank networks fail.\nSimplicity:\n Users appreciate that our tools "just work" without unnecessary flashy features.\nAreas for Improvement\nCustomer Support:\n There is a perception of delayed responses. For a rural user, a delayed resolution immediately erodes trust.\nDispute Resolution:\n Handling failed transactions and "hanging" funds remains a critical pain point.\nUser Feedback Loops:\n We need to listen to what users are doing (e.g., uninstalling the app) as much as what they say on social media.\nTroubleshooting & Discussion\nDuring the session, several key questions arose regarding how we apply these theories.\nQuestion:\n How do we reach the unbanked who don't trust banks?\nAnswer:\n Leverage the device they already trust: their phone. As of 2025, mobile penetration in Nigeria is at 87%, higher than the banked population. Solutions like M-Pesa succeeded by starting with basic USSD text services rather than complex apps.\nQuestion:\n How can we improve UX for low-literacy users? \nAnswer:\n Move beyond text. Research found that "Amina" (the rural trader persona) could easily navigate tricky interfaces when voice prompts were added.\nQuestion:\n Is technology always the solution? \nAnswer:\n No. Sometimes a user stops using an app simply because they don't understand the loan terms, not because the app is buggy. This is a communication gap, not a code bug.\nConclusion\nProfessor Afolabi left us with a striking calculation: If we capture 40 million users and earn just ₦100 per user per day, the revenue impact would be massive but only if we retain their trust.\nUser modelling is not just an academic exercise; it is the path to financial inclusion. By understanding the "Amina" in Wasimi and the "Chinedu" in Lagos, we can build products that are not just functional but essential. As we continue to refine our engineering roadmap, let’s remember: "Listen to the customer... Listen to what they are saying and what they are not saying".\nYou can watch the session \nhere\n

Data Connections For Fintech Services

Moniepoint Research Talks is our monthly engineering chat, where research meets practice. Each month, we host academics and senior engineering experts to share insights that help engineers build better systems, solve real-world problems, and advance their craft.\nIn this edition, Dr Oladipupo Sennaike from the Department of Computer Sciences at the University of Lagos (UNILAG), for a recent instalment of the Moniepoint Research Talks. Dr Sennaike presented his research on automatically discovering connections within vast collections of public data, demonstrating a robust method for improving how we interact with open data globally.\nThe Context: Why Open Data Portals Matter\nOpen data portals are online platforms that provide access to collections of open, freely available data intended for reuse. Globally, there are over 2,600 such portals, including major examples such as data.gov for the U.S. government and data.gov.uk for the U.K., as well as the Nigeria Data Portal and the World Bank Open Data platform.\nThese portals are critical for several reasons:\nTransparency & Accountability:\n They enable citizens to access information on government spending, budgets, and public projects.    Cxxzxxz z\nInnovation & Entrepreneurship:\n Developers and startups leverage open data to build new fintech, transport, health, and civic-tech products.\nResearch & Education:\n Academics and analysts use these datasets for modelling, policy research, and AI training. Furthermore, data collected from these portals provides the evidence needed to ensure that policies are based on real-world evidence.\nBetter Public Services:\n Governments can improve planning for sectors like transport, health, and agriculture by harnessing insights derived from this data.\nOpen Data Portals generally operate on a generic three-layer architecture: the \nUser Interface \n(containing the Catalogue UI, Search UI, Analytics, and APIs), \nServices \n(handling the data catalogue, metadata management, and search), and \nStorage \n(managing files, structured data, and indices).
⁠\n\n
⁠The Core Problem: Current Limitations in Search\nDespite their importance, existing open data platforms offer only basic search capabilities and simple filtering. A critical limitation is the \nlack of recommendations for related datasets\n. Understanding which datasets are related (or semantically connected) is essential for users who want to integrate or "mash-up" data for comprehensive analysis or to create new data-driven services.\nGiven the scale of these catalogues data.gov, for example, hosts over 350,000 datasets, and data.gov.uk has over 55,000, manually specifying the relatedness relationships across them is simply infeasible.\nThe Research Objective: Automating Semantic Relatedness\nDr. Sennaike’s primary goal was to determine the \nimplicit semantic relatedness\n of datasets published within a catalogue. The proposed solution was to use a Self-Organising Map (SOM) to generate automated dataset recommendations during search operations.\nThe work acknowledged other established methods for measuring relatedness, such as Lexical resource-based approaches, Explicit Semantic Analysis (ESA), and Latent Dirichlet Allocation (LDA) for topic modelling. Still, it focused on the SOM's unsupervised capabilities.\nLeveraging Kohonen’s Self-Organising Maps (SOM)\nA \nSelf-Organising Map (SOM)\n is a type of unsupervised artificial neural network. Unlike supervised learning, where a model is trained on labelled data (and its accuracy is easily measured by comparing its output to the correct label), unsupervised learning can learn connections from unlabeled data. The SOM acts similarly to clustering algorithms by grouping similar concepts, but its clusters do not have hard boundaries.\nThe core functionality of the SOM algorithm is dimensional reduction: it projects high-dimensional input data (with many attributes or features) onto a low-dimensional space, typically a two-dimensional map, while preserving \ntopological order\n. This preservation means that \nrelated data points are placed close to one another on the resulting map\n.\nThe SOM consists of nodes or units arranged in a regular rectangular or hexagonal grid. Each node is associated with an n-dimensional \nmodel vector\n (or weights) that approximates the set of input data, where \nn\n is the dimension of the input space (the number of features).\nDuring training, data items are presented to the nodes in parallel. The nodes compete, and the node with the \nbest-matching model vector, usually determined by\n the Euclidean distance metric, emerges as the winner. The model vectors (weights) of this winning unit and its neighbours are then adjusted (or strengthened) to move closer to the input data, enabling learning. This adjustment to the surrounding nodes is critical, as it defines the neighbourhood size and helps preserve the topological order.\n\n
⁠\nThe SOM Model Development Process\nThe research utilised a five-step methodology to implement the SOM-based recommendation engine:
⁠\n\n
⁠\n1. Data Preparation and Extraction\nThe study focused on 255 open datasets from the \nDublin City Open Data Platform (DubLinked)\n. This catalogue includes diverse information ranging from pedestrian footfall indices to commercial lease registrars and sculptures in public parks.\nData was extracted from the DubLinked CKAN platform instance using its \nREST API\n. The extraction captured essential metadata, including the dataset's \nTitle\n, \nOrganisation\n, \nTheme\n, and \nTags\n. Furthermore, \nNamed Entity Recognition (NER)\n was employed to extract specific entities, such as people, organisations, and locations, from the content and metadata, thereby enriching the dataset description.\n[TABLE: Extracted Dataset Features, including Title, Organization, Theme, Notes, Tag, Resource Fields, Location, Person, and Organization]\n2. Data Transformation\nSince the SOM requires numeric vector input, the text-based metadata and features were transformed into a \nDocument Term matrix\n using the Term Frequency–Inverse Document Frequency (\ntf-idf\n) metrics. This common NLP technique converts variable-length text documents into a fixed-size numeric representation, ensuring regularity across all datasets. The resulting input matrix for the SOM was 255 by 1241 (representing the 255 datasets and 1,241 terms/features extracted).\n3. Model Selection, Validation, and Results\nSince there was no labelled data to rely on, the researchers experimented with different map sizes to find the optimal SOM configuration. Selection was based on two key metrics:\nTopological Error:\n This measures the proportion of input samples in which the first- and second-best matching units are \nnot\n neighbours on the map. A low error suggests that topologically close items on the map are genuinely similar in the source data.\nQuantisation Error: \nThis is the average distance between the input data and the model vector of its best-matching unit. This indicates how accurately the map represents the overall training data.
\n\nFor validation, the process relied on \ndomain experts\n who examined the categories discovered by the map. A category was defined by the data items falling onto a specific node, plus those within a specified radius around that node. Across all testing cases, these experts successfully inferred the underlying concept addressed by each map category, even when the neighbourhood radius was increased.\nThe resulting SOM visualisations revealed clear groupings of semantically related datasets:\n\n\n
⁠\n\n
⁠Further analysis using \nWord Clouds \nbased on the metadata and node features helped visualise the concepts captured by clusters of datasets. For example, one node might clearly display keywords such as Culture, Heritage, and Arts, confirming that the topic is being grouped.
⁠
⁠\n\n
⁠\nApplication and Conclusion\nThe developed SOM-based model effectively computed semantic relatedness among datasets, providing strong evidence for the efficacy of the unsupervised approach. This method reveals opportunities for innovation implicit in the data catalogue by making hidden connections explicit.\nWhen incorporated into a search interface, the model generated highly relevant recommendations:\nA user searching for the "\nLuas Stops\n" dataset (Dublin’s tram network) received related resources like "\nTraffic Volumes\n," "\nReal-time passenger Information (RTPI) for Dublin Bus\n," and "\nCommercial Bus Services in Ireland\n".\nA search for the "\nParks\n" dataset yielded connections to "\nArt in the Parks\n," "\nDLR Landscape Maintenance\n," and "\nTrees\n".\n\n
⁠\nChallenges and Future Work\nWhile the SOM approach proved robust, the research identified ongoing challenges. Specifically, poor-quality metadata and data could negatively affect the model's effectiveness, as it relies heavily on textual descriptions and entity recognition extracted from this metadata.\nFuture work will focus on integrating the model with other tools to enable the seamless integration of compatible datasets and applying the model to much larger-scale data catalogues beyond the initial 255 used in this study.

Discovering connections in open datasets

Introduction\nMoniepoint Research Talks is our monthly engineering chat, where research meets practice. Each month, we host academics and senior engineering experts to share insights that help engineers build better systems, solve real-world problems, and advance their craft.\nIn this edition, \nDr\n \nIshaya Gambo\n, a leading researcher from \nObafemi Awolowo\n University, shared his research on "Redefining Stakeholder Engagement in Software Engineering”. His insights forged through years of research, including during his master's and PhD, focused on resolving conflicts among software development stakeholders, challenging our conventional views on development and requirements gathering.\nAt the heart of Dr Gambo’s message is a fundamental truth: \nEvery successful system runs on two APIs: the machine API and the human API\n. While technical APIs define how systems communicate, \nHuman APIs define how people understand each other\n. When the Human API breaks, even if the code is perfectly clean, the project is destined to fail.\nThis is why, at Moniepoint, we are moving beyond passive requirement collection towards a model of continuous relationship-building, treating requirements as living conversations.\n\n\n
⁠\n
⁠The Challenge We Face in Fintech\nMoniepoint operates within a highly intricate ecosystem. Every product decision impacts a diverse array of influential stakeholders. This complexity often leads to conflicting demands when launching a new feature, such as a payment service:\nThe Business Team:\n Often wants speed and revenue generation, expressing the need for the product "yesterday".\nThe Customers (Individuals & SMEs):\n Demand simplicity, ease of use, reliability, and unwavering trust.\nThe Compliance Team:\n Is primarily interested in security and ensuring regulatory rules (like those set by the CBN) are not compromised.\nThe Developers/Engineers:\n Need a clear set of requirements to deliver the system successfully.\nIn this competitive fintech landscape, where \ntrust is the primary currency\n, poor stakeholder engagement directly leads to project delays, costly rework, and lower user satisfaction.\n
⁠Moving Beyond the Traditional Approach\nThe traditional software development model often follows a rigid, linear path: \nGather Requirements -> Build -> Deploy\n. Dr. Gambo highlighted that this waterfall-like rigidity inevitably leads to crises:\n
⁠The Gaps of Traditional Software Development\nThe critical element missing from the conventional approach is \ncontinuous relationship building\n. This absence creates three major problems:\nConflicting Requirements:\n Requirements gathered from different stakeholders frequently conflict, leading to reactive compromises late in the process.\nLate-Stage Changes:\n Development is often disrupted when stakeholders realise midway through the project that the solution doesn't align with their true vision, leading to scope creep and delays.\nStakeholder Dissatisfaction:\n Even when the final product technically meets the documented requirements, stakeholders remain unhappy because we addressed their \nstated wants\n instead of their \nunderlying needs\n.\n
⁠A New Perspective: Requirements as Conversations\nTo overcome these gaps, \nDr. Gambo\n proposes a new perspective rooted in research, viewing requirements as ongoing conversations built on deeper inquiry and trust:\n
⁠1. From Collecting to Engaging\nWe must shift from passively collecting requirements as static artefacts to actively \nengaging\n with stakeholders to discover needs through collaborative exploration. This involves proactive interaction, such as engineers sourcing feedback directly from Moniepoint customers or holding cross-functional brainstorming sessions.\n
⁠2. From Static Documents to Ongoing Dialogue\nA requirement must not be treated as a document to be signed off or frozen. Instead, we must treat it as a \nliving conversation\n that evolves as our collective understanding of the problem deepens and new insights arrive.\n
⁠3. From ‘What’ to ‘Why’\nThe focus of our inquiry must move beyond \nwhat\n the stakeholder wants (e.g., "faster payment process") to \nwhy\n they want it. Unveiling the 'why' reveals the true underlying business or user problem, which is crucial because a requirement defined by one stakeholder might mean something entirely different to another.\n
⁠4. From Transaction to Trust\nThe ultimate goal of stakeholder engagement should be the \ncultivation of trust\n, rather than a simple transactional exchange of information. This trust makes all subsequent collaboration, innovation, and relationship management more efficient and effective.\n
⁠The Three-Pillar Framework for Better Engagement\nDr. Gambo presented an evidence-based framework for managing the natural conflicts that arise around competing objectives like \nSecurity, Speed, and User Experience (UX)\n. The solution lies not in avoiding conflict, but in how we handle it. The framework consists of three pillars:\n
⁠Pillar 1: Identify\nWe must accurately determine who all the stakeholders are and what they truly need. This means going beyond job titles to understand their roles and motivations.\nAction:\n Employ \nStructured Interviews\n, \nGoal Mapping\n, and \nPersona Development\n.\nKey Distinction:\n Identify both \nfunctional requirements\n (what the system should \ndo\n, e.g., logging in) and the more challenging \nnon-functional requirements\n (what the system should \nbe\n, e.g., fast, secure, usable).\n
⁠Pillar 2: Prioritise\nNot all requirements are equal. We must use a structured approach to choose which requirements deliver the most value.\nCriteria:\n Prioritise based on \nBusiness Value, User Impact, Technical Feasibility,\n and \nCompliance\n.\nResearch Strategy:\n An effective prioritisation strategy, such as the MEO model (Mandatory, Essential, Optional), helps determine which features must be implemented immediately, which are secondary, and which can wait for subsequent releases.\nMethods:\n Practical prioritisation methods include \nWeighted Scoring\n (assigning weights to strategic factors) and \nCollaborative Ranking\n (facilitated sessions in which stakeholders discuss trade-offs). Hybrid models combine quantitative data with qualitative discussion.\n
⁠Pillar 3: Resolve\nWhen stakeholders disagree, the focus must be on \nstructured dialogue, not debate\n.\nTechniques:\n Approaches include \nFacilitated Workshops\n (using a neutral facilitator), \nTransparent Decision Criteria\n (agreeing on criteria before debate starts), and \nTrade-off Analysis\n.\nThe Goal:\n The aim is informed compromise, or \nGoal Consensus\n, where all stakeholders understand the decision, feel their input was valued, and commit to the path forward, not simply creating "winners and losers".\n
⁠Practical Takeaways for Moniepoint\nTo embed this research-driven mindset across our engineering teams, Dr. Gambo provided specific, actionable advice:\nInvest Time in Stakeholder Mapping:\n Know the ecosystem intimately, including regulators, customers, merchants, and internal teams, to understand their diverse needs and perspectives.\nCreate Regular Touchpoints:\n Establish frequent points of engagement (e.g., monthly or bi-annual field work) with key stakeholder groups, such as Moniepoint customers or POS individuals, to gather deep insights and information that were previously unconsidered.\nUse Structured Methods for Prioritisation:\n Avoid relying solely on intuition; use structured, research-backed methods to prioritise requirements and manage resources effectively.\nAddress Conflicts Early:\n Utilise facilitated conversations to resolve conflicts proactively.\nDocument Decisions and Rationales Transparently:\n Ensure all decisions and their justifications are clear and accessible to build trust.\n
⁠Building a Culture of Engagement\nUltimately, Moniepoint's engineering future is human-centred. This shift is not merely about implementing new processes; \nit is about building a mindset\n.\nA culture of engagement means recognising that the product, process, and \npeople\n are the three key pillars of software engineering. We must foster an environment that:\nEncourages cross-functional collaboration.\nCreates safe spaces for voicing concerns, where every perspective is valued.\nCelebrates stakeholder partnerships and learns from every project.\nBy investing in relationships, we see measurable business impacts, including \nFaster Time-to-Market\n, \nReduced Project Risks\n (since conflicts are addressed early), and \nHigher User Satisfaction\n (because true needs are met).\nAt Moniepoint, every feature we build connects a network of stakeholders, and these relationships are our true architecture. When we embed empathy and evidence-based engagement in engineering, we don't just build systems, \nwe build significance.

From Requirements to Relationships: How Moniepoint Engineers are Redefining Stakeholder Engagement

\n\n
Introduction\nMoniepoint Research Talks is our monthly engineering chat, where research meets practice. Each month, we host academics and senior engineering experts to share insights that help engineers build better systems, solve real-world problems, and advance their craft.\nIn this edition, we welcomed Dr Victor Odumuyiwa, Research Lead at the Machine Intelligence Research Group at UNILAG. In his presentation, he explored how transformer models are applied to financial texts to uncover customer insights.\nThe fundamental challenge in leveraging machine intelligence is semantics. How do we ensure machines understand not just the text, but the context surrounding that information? A vast array of possibilities opens up if a machine can truly understand human communication.\nFor organisations dealing with high volumes of customer interactions, this context is crucial. Natural Language Processing (NLP), a subfield of Artificial Intelligence (AI), focuses on programming computers to process and analyse human language data. This guide will detail how Transformer models, the current state-of-the-art in NLP, can be applied to financial texts to automatically generate actionable insights.\n
⁠Prerequisites / What You'll Need\nFamiliarity with foundational AI/ML concepts (RNN, LSTM, Neural Networks).\nBasic understanding of NLP tasks (Classification, Translation, Generation).\nKnowledge of metrics used for model performance evaluation (Precision, Recall/Sensitivity, Accuracy).\n
⁠The Foundation: The Rise of the Transformer Architecture\nThe journey toward contextual understanding evolved through several stages: starting with Recurrent Neural Networks (RNN), moving to Long Short-Term Memory (LSTM), and culminating in the \nTransformer\n architecture.
\nA key limitation of earlier models like LSTM was the difficulty in maintaining context over long sequences (the vanishing gradient problem), even with improvements like Bi-LSTM (processing in both directions).\n\nThe breakthrough came in 2017 when Google published the paper, \n"Attention is all you need"\n. Attention mechanisms changed the flow of NLP by enabling the model to remember and connect words to past words, understanding relationships across the entire sequence, which is critical for human language comprehension.\n
⁠Core Components of the Transformer\nThe Transformer relies on several mechanisms to capture semantic meaning and positional context:\nTokenisation and Embedding:\n Input text is broken down into numerical tokens (words or subwords), which are then converted into numerical vectors (embeddings) to capture their semantic meaning.\nPositional Encoding:\n Since the model processes input in parallel, vectors are added to the input embeddings to provide information about each token's position, ensuring the model understands the order of the sequence.\nSelf-Attention Mechanism:\n This is the core component that determines the relationship between all tokens in a sequence simultaneously, using three vectors for each token:\nQuery (Q): Represents the current token being processed.\nKey (K): Acts as a label for every other token in the sequence.\nValue (V): Contains the actual content of the tokens. By computing attention scores between the Query and all Keys, the model learns how much attention to pay to each Value, forming a weighted, contextual representation.\nBERT (Bidirectional Encoder Representations from Transformers):\n Introduced in 2018, \nBERT\n further refined this by processing text in more directions, leading to better contextual representations. This model started the trend of predicting the next word using masked input, enabling effective content generation.\n
⁠Key NLP Tasks in Financial Services\nThese advanced models are applied across several critical NLP tasks, moving far beyond simple text processing to semantic understanding.\n
⁠Sentiment Analysis (Customer Opinion)\nSentiment analysis determines the opinion expressed in a piece of text (e.g., positive, negative, neutral, angry, or hateful). This is essential for understanding customer intimacy and public perception, especially regarding new products or services.
\nIn testing various models on financial datasets (including customer complaints and financial news), \nBERT\n consistently showed better \nAccuracy, Precision\n, and \nRecall\n results compared to traditional machine learning models like SVM (using TF-IDF) and older neural networks like LSTM. For example, on one financial sentiment analysis dataset (Dataset 2), BERT achieved 83% accuracy, significantly outperforming SVM (68%) and LSTM (70%).\n
⁠Financial Named Entity Recognition (NER)\nNER automatically identifies a document's names and specific entities (like countries, people, or organisations). This capability is critical for relationship extraction and synthesising information from millions of business news articles and documents, giving financial professionals a concise summary and insight.\n\nNER models, using optimisers like AdamW and loss functions like Cross-Entropy, can accurately tag specific entities in sentences, whether using BERT or Llama models.\n
⁠Machine Translation\nSignificant progress has made machine translation highly effective today, unlike the difficulties encountered years ago. Systems leveraging Transformer architectures (like Google's and OpenAI's models) can now translate complex financial documents, such as news concerning an $800 million World Bank grant to the Nigerian government, moving beyond literal translations to retain the general literary meaning.\n
⁠Financial Question Answering (QA)\nFinancial QA involves training a model on specific data, such as company financial reports (10-K filings). The trained machine can then accurately respond to particular questions, reducing the need for humans to handle routine queries. Models like \nRoBERTa\n have been trained to answer questions regarding financial details, such as the value of a one-time transaction tax.\n
⁠Generating Actionable Customer Insights\nThe true value of these NLP tasks lies in combining them to generate comprehensive customer insights, rather than treating them as scattered tasks.\nA system can take unstructured data (like a customer complaint), perform several tasks such as \ncategory detection\n (e.g., Debt collection), \npriority detection\n (e.g., Urgent), and \nsentiment detection\n (e.g., neutral) and then plug this classified information into an LLM or Transformer to generate detailed summaries.\nExample Insight Generation from a Customer Complaint: Based on a complaint detailing an unrecognised debt, an unauthorised account levy, and communication failures due to an outdated address, the combined system generates multi-faceted insights:\nDisputed Debt Handling Issues:\n The required process (affidavit of fraud, police report) was overly burdensome, leading to unresolved disputes.\nCommunication and Notification Failures:\n Legal correspondence was sent to an outdated address, despite the company having the correct, current address on file from a recent payment.\nData Management Concerns:\n This situation reveals a clear breakdown in maintaining and cross-referencing up-to-date customer information, suggesting poor data management.\nCustomer Impact:\n The resulting judgment caused a financial loss ($830 levied) and significant emotional distress for the customer.\nFinally, the system suggests \nProcess Improvement Opportunities\n and \nActionable Insights\n, such as implementing robust address verification protocols before legal actions and simplifying the debt dispute process.\n
⁠Troubleshooting & FAQ\nQuestion:\n When is it most efficient to fine-tune an LLM versus prompt engineering?\nAnswer:\n Fine-tuning and prompt engineering serve different goals, but both depend heavily on the context.\nFine-tuning\n is primarily used to adapt a general model (pre-trained on broad language) to a \nspecific domain\n or form (e.g., financial jargon, JSON output format). You are modifying the model’s parameters to understand your particular data better. For fine-tuning to be effective, you need a substantial dataset—hundreds of thousands of examples, as smaller datasets may yield inaccurate results.\nPrompt Engineering\n (or prompt tuning) helps provide specific \nfacts\n or dictate behaviour, but the size of the context window limits it. If your context is extremely large (e.g., many data sources), prompt tuning may not be viable due to token limits.
\nQuestion:\n For Financial Question Answering, should we use Retrieval-Augmented Generation (RAG) or train from scratch?\nAnswer:\n While you can train a small language model entirely from scratch if you have sufficient data and compute power, RAG is generally recommended for integrating real-time or proprietary financial data.\n
\nRAG augments the knowledge of a pre-trained model (which is limited by the information available during its training, perhaps six months ago) by fetching current, external, or private information. This is necessary to integrate new data into the model’s knowledge base without retraining the expensive base model every day. RAG also provides a mechanism to allow an LLM to interact with sensitive resources, like your internal database, by creating a system that extracts context-specific information for generation, while keeping the database secure.\n
⁠Conclusion: The Road Ahead for African AI\nTransformer models provide unprecedented power for extracting insight from text. However, the future of AI in regions like Africa requires a focus on AI localism.
\nSince most African languages are spoken rather than written, focusing on multimodal data (spoken and acoustic data) is essential to drive financial inclusion. Currently, most African languages (like Yorùbá, Igbo, and Hausa) are poorly represented in global AI tools due to factors like the absence of standardised orthographies and the paucity or absence of digital corpora (web-scale data).\n\nTo maximise the benefits of AI for African populations, there must be intentionality and strong collaboration between industry, academia, and philanthropy. The strategic direction should focus on building and using Small Language Models (SLMs) tailored to local contexts, rather than trying to compete in the race for Artificial General Intelligence (AGI). This focus will ensure that ordinary people, such as small business owners, can benefit from AI by receiving truly contextual and relevant advice.\n

Unlocking Customer Insight: Applying Transformer Models to Unstructured Financial Text

\n\n
Introduction\nMoniepoint Research Talks is our monthly engineering chat, where research meets practice. Each month, we host academics and senior engineering experts to share insights that help engineers build better systems, solve real-world problems, and advance their craft.\nIn this edition, Comfort Lawal from Covenant University shared her research on the use of federated graph neural networks to detect anomalies in the protected systems. \nAccording to her, complexity is the new normal. Enterprises rely on multi-cloud architectures (used by 73% of organisations) that generate an astounding 500+ billion logs daily. This massive, distributed infrastructure poses a severe challenge to security and reliability, especially given that the global annual cost of cybercrime has reached €1.75 trillion. Detecting critical system failures or malicious activity requires an immediate response, often less than 1 second.
⁠\nTraditional anomaly detection methods, such as centralised monitoring and isolated rule-based systems, fail to meet this challenge. They often suffer from privacy violations, bandwidth bottlenecks, high false positive rates, and an inability to adapt to new threats.\n
⁠The Problem with Traditional Approaches\nTraditional security and monitoring methods fall short in these distributed, high-volume environments.\nCentralised Monitoring:\n Fails due to privacy violations when raw data is pooled, creating single points of failure and causing bandwidth bottlenecks.\nIsolated Detection (Classical ML):\n Treats data as independent, leading to high false positive rates. For instance, a poor network connection might be falsely flagged as an anomaly because the system fails to check the related data context.\nRule-Based Systems:\n Cannot adapt to new or evolving threats and require constant, manual maintenance.\n\nThe Solution:\n Converging Technologies\nThe Federated Graph Neural Network (F-GNN) framework is an innovation designed to overcome these critical limitations by combining three powerful technologies:\nGraph Neural Networks (GNNs):\n Capture complex relationships in vast network data, which is a natural fit for cloud systems.\nFederated Learning (FL):\n Allows models to be trained without sharing raw, sensitive data, enabling privacy-preserving collaboration across distributed sites.\nHyperbolic Geometry (HGNN):\n Provides an efficient way to model the hierarchical nature of cloud architectures.\n
⁠Prerequisites / What You'll Need\nDeploying and experimenting with F-GNNs requires expertise and infrastructure across several domains:\nExpertise:\nMachine Learning Engineers with GNN expertise.\nSecurity Engineers experienced in privacy protocols.\nDevOps Engineers for infrastructure management.\nCore Frameworks:\nFlower: Recommended framework for Federated Learning.\nPyTorch Geometric (pyg.org): Used for GNN implementation.\nDGL (dgl.ai): Crucial for distributed graph processing, especially when dealing with large graphs (e.g., 66,000+ nodes and millions of edges).\n
⁠Core Concepts and Technical Deep Dive\nFoundational Concepts:\n Understanding the Building Blocks\nGraph Neural Networks (GNNs)\nCloud systems are a natural fit for graph representation. In this context:\nNodes represent components like Servers, VMs, or Containers.\nEdges represent relationships such as Network connections, Dependencies, or API Calls.\nFeatures include metrics, logs, and system states.\nThe key advantage of GNNs is Message Passing, where nodes learn from their neighbourhood to reduce false positive rates and solve the challenge of isolated detection by considering underlying relationships.\n
⁠Common GNN architectures include:\nGraph Convolutional Networks (GCN):\n Fast and efficient, suitable for static graphs, but perform transductive learning, requiring retraining for new data.\nGraph Attention Networks (GAT):\n Uses an attention mechanism for dynamic importance, often used for threat detection, but it is of higher complexity.\nGraphSAGE:\n Supports inductive learning through scalable sampling, making it suitable for large and constantly changing networks.\n
⁠The Hyperbolic Advantage\nWhen dealing with deeply hierarchical data—such as the structure of a cloud environment (Data centre → Cluster → Node hierarchies)—Euclidean space introduces problems like poor hierarchy preservation and high dimensions. Hyperbolic Geometry naturally caters to these hierarchies, offering:\nNatural hierarchy embedding.\nEfficiency: Requiring 10x fewer parameters.\nPerformance Gain: Demonstrated 16-63% performance gains in the literature.\n
⁠Federated Learning (FL)\nFL allows the training of a Global Model continuously improved by training on decentralised, local graph data residing on different cloud sites that form a federation. This process involves:\nTraining models without sharing raw data.\nEnsuring privacy-preserving collaboration.\n
⁠The Innovation: The Federated GNN Architecture\nThe F-GNN framework operates via a continuous learning cycle coordinated across a decentralised network.
\n1. Local GNN Training\nEach participating cloud centre or edge server trains a GNN model locally using its site-specific data and logs. This layer ensures that raw data privacy is preserved. The local GNN calculates an anomaly score for each node.
⁠\n2. Secure Aggregation\nAfter local training, only the model updates (gradients) are sent to the global coordination server, not the raw data. These updates are protected using advanced privacy techniques:\nPrivacy Technique\nFunction/Guarantee\nDifferential Privacy\nAdds noise injection to the model updates to mask specific information, providing mathematical privacy guarantees\nHomomorphic Encryption\nAllows computation on encrypted data (full privacy) and can involve quantum-resistant options\nSecure Aggregation\nUses secret sharing protocols and Byzantine fault tolerance to protect the combined model updates\nCrucially, implementing security involves a performance vs. privacy trade-off. Testing showed that while full encryption (Homomorphic Encryption) achieved high privacy, accuracy dropped to 85%. Using Differential Privacy (medium privacy) yielded an accuracy of approximately 97%.
\n3. Global Model Distribution\nThe model updates are securely aggregated (often using the FedAvg algorithm with enhancements like gradient clipping) to create an improved global model. This global model is then distributed back to all local sites, completing the continuous learning cycle. The global coordination server can sit in any one of the federating cloud providers.\nMulti-Layer Privacy Protection\nTo secure the model updates transmitted between local clients and the global server, the framework employs multi-layer protection techniques:\nDifferential Privacy (DP):\n Provides mathematical privacy guarantees by adding noise injection to the model updates, masking the information.\nHomomorphic Encryption (HE):\n Allows computation on encrypted data, offering full privacy and quantum-resistant options, although it can significantly impact model accuracy.\nSecure Aggregation:\n Uses secret sharing protocols and Byzantine fault tolerance to ensure secure computation on the updates.\nPerformance Breakthroughs\nImplementing these security measures must not negate the necessity for real-time response. The FGNN framework achieves groundbreaking efficiency through several optimisation techniques:\nCommunication Optimisation:\n The system employs Asynchronous Updates (where clouds send updates based on customised rules, like volume or accuracy changes), Top-k sparsification, and Gradient compression. This optimisation, achieved through methods like FEDZIP, resulted in a 97.6% bandwidth reduction. Communication time dropped from 700 seconds per round to 16.8 seconds per round.\nModel Compression:\n Techniques like Quantisation (reducing updates from 32-bit to 8-bit without information loss), Pruning (removing unnecessary parameters, achieving 50% removal), and Knowledge Distillation (where a faster student model learns from a robust teacher model) provide a 10x speedup and ensure production readiness.\n\nReal-World Impact: Results That Matter\nThe FGNN framework provides high Privacy+Performance.\nMetric\nRange/Value\nInsight\nDetection Rate\n85.02% - 99.998%\nAccuracy for DoS attacks is near-perfect (99.998%).\nLatency\n< 100ms\nEnables true real-time critical alerts.\nCommunication Reduction\n97.6%\nBreakthrough in bandwidth optimisation.\nEnergy Preservation\n99%\nSignificant efficiency gains.\nPrivacy Cost\n12% loss in accuracy\nFull encryption (High Privacy) reduces accuracy from 99.6% (No Protection) to 87.85%.\n\nKey deployment sectors include:\nFinancial Services:\n Used for cross-institution fraud detection, maintaining 99.2% accuracy with zero data breaches.\nCloud Providers:\n Applicable for cost anomaly detection (AWS), cognitive service integration (Azure), and edge deployment readiness (GCP).\nHealthcare:\n Enables distributed patient monitoring and HIPAA-compliant, cross-hospital learning.\nTroubleshooting & FAQ\nQuestion:\n Traditional models often suffer from poor explainability. How does this deep learning approach provide sufficient justification for an anomaly?\nAnswer:\n Explainability is primarily achieved through hierarchical classification, which goes far beyond a simple "anomaly/not anomaly" output. The system breaks down the alert into different levels:\nDetection: An anomaly occurred.\nType of Anomaly: Is it a critical anomaly (system about to fail), an error anomaly, or a warning anomaly (requires different actions)?\nApplication: Which cloud application produced the anomaly (e.g., computing, messaging, networking)?\nSpecific Model/Component: Pinpoints the exact component causing the issue, allowing administrators to target their troubleshooting efforts.\nThis hierarchical breakdown makes the system more transparent, although it currently stops short of explaining the ultimate "why" behind the specific component failure.\n
\nQuestion:\n Are there standard comparative works that show whether Homomorphic Encryption or Differential Privacy is definitively "better" for F-GNN implementation?\nAnswer:\n To the best of current knowledge, a standard work comprehensively comparing all privacy or optimisation algorithms in a federated graph learning context does not yet exist. However, practical experimentation revealed a significant trade-off: Homomorphic Encryption (offering complete privacy) caused the accuracy to drop to 85%, while Differential Privacy (medium privacy) maintained roughly 97% accuracy. Choosing the correct protocol depends on whether the organisation prioritises maximum privacy or performance.
\nQuestion:\n When comparing F-GNN to centralised or classical models for anomaly detection, how much better is the federated graph approach?\nAnswer:\n Comparative analysis showed that the F-GNN model provided a 15.7% improvement in accuracy compared to existing anomaly detection works, including centralised and flat classification approaches. Furthermore, the F-GNN model is built with the intelligence to automatically tweak its parameters (like learning rates) based on local data characteristics. It is crucial for solving convergence challenges associated with non-Independent and Identically Distributed (non-IID) data in federated environments.\nConclusion\nThe Federated Graph Neural Network framework represents a pivotal step forward, enabling organisations to address massive data volumes and critical, sub-second latency requirements while strictly adhering to privacy protocols.
\nBy leveraging GNNs, Federated Learning, and the efficiency of Hyperbolic Geometry, this approach achieves 99%+ accuracy while maintaining complete privacy compliance. The breakthrough communication efficiency, reaching over 97% communication reduction, makes true, real-time federated learning viable.\n
⁠Next Steps and Future Vision\nThe journey toward fully optimised, secure AI is ongoing. Organisations and practitioners should:\nInvest in Privacy-Preserving Technology:\n Prioritise technology that maintains regulatory compliance while offering a competitive advantage through collaboration.\nExplore Novel Architectures:\n Researchers are encouraged to explore advanced areas like hyperbolic GNN variants and hybrid approaches (e.g., Transformer-GNN fusion) to further improve detection rates, especially since theoretical work suggests HGNN can dramatically simplify hierarchical classification.\nFuture Integrations:\n The framework is built to be future-proof, with planned integration for Large Language Models (for natural language anomaly explanations) and Quantum Computing (for ultra-low latency processing).\n

Federated Graph Neural Networks: Solving Real-Time Anomaly Detection Without Sacrificing Privacy

Introduction\nMoniepoint Research Talks is our monthly engineering chat, where research meets practice. Each month, we host academics and senior engineering experts to share insights that help engineers build better systems, solve real-world problems, and advance their craft.\nIn this edition, Dr Ernest E Onuiri shared his knowledge of precision medicine and his learning from machine learning for cancer detection. The promise of healthcare often feels like a distant ideal, especially when faced with the "one-size-fits-all" approach to treatment. Imagine receiving medicine for malaria that works for some, but not for you, or a cancer treatment that cures one patient but harms another. This disparity highlights a fundamental problem that precision medicine aims to solve: tailoring medical interventions to the individual. But while the vision of customised care is compelling, its widespread adoption faces significant challenges, particularly around scalability, data complexity, and a collective reluctance to embrace change. We will explore the core concepts of precision medicine, delve into the obstacles hindering its scalability, and propose pathways for collaboration between industry and academia to build the trustworthy AI systems needed to revolutionise healthcare.\n
⁠What is Precision Medicine?\nAt its heart, precision medicine is about moving beyond generic treatments to customise care for each individual. The goal is to minimise adverse drug reactions and autoimmune problems that can sometimes be more harmful than the disease itself. This personalised approach relies on a rich collection of data points, including:\nClinical Records:\n Observable features like temperature, weight, and responses to basic questions gathered during a hospital visit. These are generally easy to understand.\nGenomic Landscape:\n The complex internal data within humans often explains why things go wrong internally before external manifestations appear. This data is not easily accessible, expensive to obtain, and requires specialised tools and expertise to interpret, often appearing like noise to the untrained eye.\nImaging Data:\n Various forms of visual data obtained through technologies like MRI, CT, ultrasound, or X-ray.\nSeveral key principles underpin precision medicine:\nPersonalisation:\n Customising medicine for a patient based on their specific needs.\nPrediction:\n The ability to anticipate health events, which enables proactive prevention and treatment.\nParticipation:\n Empowering the patient to participate actively in their healthcare journey, asking questions, understanding the proposed treatments, and giving a sense of ownership.\n
⁠The Scalability Gap: Challenges in Precision Medicine\nDespite its promise, precision medicine encounters substantial hurdles in scaling beyond controlled research environments to widespread clinical adoption.\nData Complexity and Heterogeneity:\n Diseases like cancer are inherently heterogeneous, with various subtypes and severity levels. Genomic data, unlike simple clinical records, is incredibly complex and requires intermediate tools and programming languages (like R) to be unpacked and made intelligible. Combining different data modes (clinical, genomic, imaging) further exacerbates this complexity.\nData Integration and Quality:\n Healthcare data is often siloed, stored in disparate locations (e.g., Lagos, London, Tokyo) and in formats that appeal to local practices, making integration a hurdle for researchers. Furthermore, data is rarely clean, often containing missing values, noise, or being jumbled, leading to wasted research time and unreliable insights. Even long-standing repositories in institutions may not be trustworthy for new applications because the initial workflows were not designed for broad utility.\nCost and Accessibility:\n While the technology exists for genomic sequencing, it remains prohibitively expensive and is not readily available in most hospitals, limiting its widespread application.\nReluctance to Change:\n A significant challenge is the human element – the resistance to adopting new, unconventional approaches. Experienced medical professionals may adhere to traditional methods, viewing new technologies like Electronic Medical Records (EMRs) with suspicion, fearing activity tracking rather than appreciating their potential for record-keeping and knowledge transfer.\nLack of Collaboration:\n There's often a disconnect between academia and industry. Academia might pursue theoretical solutions, while the industry focuses on immediate monetizable products, which are missing opportunities for collaboration. Academic studies, especially in certain parts of the world, often lack the funding and time to generate their own clean, fit-for-purpose data, forcing reliance on external sources.\n
⁠Lessons from COVID-19 and the Importance of Validation\nThe COVID-19 pandemic offered crucial lessons, particularly regarding the need for rigorous validation and the potential of time compression in drug discovery. The rapid development of vaccines, facilitated by in silico (computational) methods, showcased how processes that once took years could be dramatically accelerated. However, this unprecedented speed also led to questions about the longevity and nature of these interventions, as evidenced by the need for multiple doses and boosters, unlike traditional vaccines.\nThis experience underscores that validation is a never-ending process. Engineers and researchers must cultivate a "inquisitive mind" that constantly asks "what if," even when a solution works perfectly. The speaker also emphasises that simplicity often trumps complexity in engineering solutions. While complex models seem more robust, simpler approaches can usually yield equally good, if not better, outcomes with less computational burden and more efficient resource utilisation. Over-complicating solutions just for the sake of it can lead to systems struggling and ultimately failing to deliver effectively.\n
⁠Pathways to Bridging the Gap: The Power of Collaboration\nOvercoming precision medicine's scalability gap demands a concerted effort and a fundamental shift towards synergistic collaboration between industry, academia, and regulatory bodies.\nIndustry-Academia Alignment:\n When industry (with its resources and real-world problems) and academia (with its research and innovative thinking) combine forces, they create something far greater than they could achieve alone. This symbiosis can translate research into monetizable solutions that address real industry needs, as seen in places like Silicon Valley's relationship with Stanford University.\nShared Data Repositories:\n Industry players, especially those generating vast amounts of data, should partner with academia to create shared data repositories. This would allow researchers to use relevant, homegrown data, leading to more tailored findings and solutions directly applicable to local contexts, rather than constantly adapting imported data and solutions.\nCustomisation in Education:\n Education itself needs to be customised to solve real problems, encouraging students to appreciate the practical application of their learning.\nLeveraging Technology for Collaborative Workflows:
\nModern technology has eliminated geographical barriers, enabling remote collaboration. This capability should be fully leveraged to foster interdisciplinary teams that bring diverse perspectives and expertise to complex problems, preventing work in silos.\n
⁠Trustworthy AI: Shared Principles Across Healthcare and Fintech\nThe need for robust, explainable, and scalable AI systems is not unique to healthcare. Industries like fintech face remarkably similar challenges and benefit from similar principles.\nRisk Prediction:\n As healthcare professionals worry about clinical risk prediction, fintech deals with financial risk prediction.\nPersonalisation:\n Precision medicine aims for personalised patient treatment; fintech strives for personalised customer recommendations.\nClassification:\n Disease subtype classification in medicine finds its parallel in customer segmentation and classification in fintech.\nFraud and Anomaly Detection:\n Protecting sensitive patient data and detecting anomalies in healthcare is mirrored by the need for financial fraud and anomaly detection, ensuring data protection and compliance.\nIn both sectors, trustworthy, explainable, and scalable systems are vital for adoption. The computational complexity often seen in precision medicine, which is a tall order, also presents a significant opportunity for innovation and scaling.\n
⁠
⁠Technology: Ability to change data into medicine\nDr. Onuiri's approach to precision medicine involves a structured workflow to transform complex, multimodal data into actionable clinical insights. Building a trustworthy learning system requires more than a robust algorithm; it demands a rigorous, multi-step process from data collection to validation.
⁠
⁠\n\n
⁠Step 1: Multimodal Data Integration and Preprocessing\nThe first major hurdle is aggregating and harmonising disparate data types into a cohesive dataset.\nExplanation:\n Data for a single patient often exists in different formats and locations, such as clinical records from a hospital's EMR, genomic data from a specialised lab, and imaging data from a radiology department. These data silos and varying standards make integration a significant challenge.\nInstruction:\n The initial step is to build a comprehensive patient profile. This involves addressing challenges like \ndata integration\n, handling \nmissing values\n, and cleaning \nnoisy data\n to ensure the quality and interoperability needed for an effective model.\n
⁠Step 2: Feature Engineering\nWith an integrated dataset, the next step is to extract meaningful signals that a machine learning model can interpret.\nExplanation:\n Raw genomic or imaging data is often too complex and high-dimensional for a model to use directly. This critical step translates the data into a structured format, identifying key \nfeatures\n or patterns—like specific gene expression levels or image textures—that correlate with particular health outcomes.\nInstruction:\n Apply domain-specific techniques to convert the raw, multimodal data into numerical or categorical features. For example, specific gene expression levels might be used as features for a cancer classification model, while texture analysis could extract features from an MRI scan.\n
⁠Step 3: Model Development and Selection\nThis is where the "learning" happens. The goal is to choose and train a model that can perform a specific task, such as predicting clinical risk or classifying disease subtypes.\nExplanation:\n While it can be tempting to use the most complex model available, Dr. Onuiri emphasises that simpler, more interpretable models often yield equally good, if not better, outcomes with less computational burden. The focus should be on creating an accurate, efficient, and explainable model.\nInstruction:\n Train various models and evaluate them based on a balance of performance, computational cost, and \nexplainability\n. In a clinical setting, a transparent model clinicians can understand and trust is often preferable to a "black box" algorithm.\n
⁠Step 4: Rigorous and Continuous Validation\nA model is only valid if it is trustworthy. Validation is not a one-time check before deployment but a continuous, iterative process.\nExplanation:\n The COVID-19 pandemic provided a critical lesson in the importance of rigorous validation. A model that performs flawlessly on a historical training dataset might fail when exposed to new, real-world clinical scenarios. Engineers and data scientists must maintain an "inquisitive mind," constantly questioning the model's performance and limitations.\nInstruction:\n Continuously test the model against new data in different clinical contexts. This ongoing validation is crucial for ensuring the system remains \nreliable, safe, and effective\n over time, forming a trustworthy AI system's bedrock.\n
⁠Conclusion\nPrecision medicine holds immense potential to transform healthcare from a generalised approach to a deeply personal and effective one. However, this future hinges on our ability to navigate the complex data landscape, overcome resistance to change, and foster unprecedented collaboration. By promoting a symbiotic relationship between industry, academia, and regulatory bodies, we can collectively plug the scalability gap, build trustworthy AI systems, and unlock the breakthroughs that have long eluded us, leading to better outcomes for everyone.\n

Designing trustworthy learning systems in precision medicine and the scalability gap

\n
Introduction\nIn this article, I share my experience contributing to the migration of a critical system from a live monolith to a new microservice, focusing on the challenges of ensuring 99.9% data accuracy before full cutover. I cover how we handled eventual consistency, validated data across two systems, and built trust through rigorous engineering practices.\nMigrating critical systems isn’t just about launching new services.
⁠\n
It’s about migrating trust. Not long ago, I was part of a team working on a major system migration: moving important business logic from a live, production monolith to a brand-new microservice with its own Postgres database. The old monolith stayed live, serving 100% of public traffic.
⁠
The new system had to prove itself — from zero — before anyone could rely on it. Our mission was clear: \nProve 99.9% data accuracy before any cutover. 
⁠\n
When we started, our accuracy was effectively 0%. This is a look back at how we approached that challenge: the strategies, the technical lessons, and how real trust is built — slowly, carefully, and with metrics.\n
⁠Setting the Stage: Monolith to Microservice - The migration involved\n•	Old System: \nA mature monolith handling millions of transactions.\n
•	New System: \nAn event-driven microservice architecture backed by Postgres.\n
•	Kafka: \nStreaming live updates and historical backfills to the new system.\n
•	Grafana + Prometheus: \nInstrumentation and monitoring for everything.\n
•	Traffic: \nStill fully routed to the monolith during migration. \n
⁠
⁠This wasn’t a lift-and-shift. It was a rebuild — while the airplane was flying.\n
⁠⁠Why Accuracy Started at Zero Syncing? \nTwo complex systems asynchronously means you start from a simple engineering truth:\nNothing is accurate until you can prove it. At the beginning:
	•	No full validation pipeline.
	•	No accuracy metrics.
	•	No live comparison in place. Without verification, assuming correctness would have been reckless.
⁠
We treated data accuracy as seriously as uptime — building it into our operational SLOs.\n
⁠⁠How We Measured Data Accuracy?\n⁠We immediately built the core metrics:\nrecords_processed_total.\nrecords_mismatch_total.\nAccuracy Rate = (processed - mismatches) / processed. \nThese were exposed through Prometheus and visualized in Grafana dashboards:\nAccuracy rate trends.\nMismatch breakdowns.\nKafka consumer lags.\nError rates across different domains. \nWe also split comparisons into:\nFresh comparisons (new events, possibly still settling).\nAged comparisons (data old enough to have converged). 
\nThis distinction kept our metrics honest and alerts meaningful.\n
⁠Challenge #1: Latency and Eventual Consistency\nBecause the new system was event-driven, eventual consistency was part of the design.\n    •	The monolith would update immediately.
	•	The microservice might lag by milliseconds or seconds (depending on event ingestion and processing). If we compared data too early, we would catch mismatches that weren’t real. \nHow We Mitigated It:\n
	•	Introduce buffer time before comparisons (e.g., 30–60 seconds).
	•	Retry mismatches before considering them final.
	•	Split fresh vs. aged metrics so stakeholders saw the full context. \nAccepting and designing around eventual consistency prevented endless debugging loops and kept trust in the migration process intact.\n
⁠⁠Challenge #2: Validating Data Across Two Systems \nValidation wasn’t a one-off — it was a continuous, ⁠multi-layered process. \nWe used two main approaches:\n   \n Shadow Reads\n
⁠	    •	Added parallel queries to both the monolith and microservice during critical transactions.
	    •	Compared key data fields silently (without affecting user-facing behavior).
	    •	Logged any differences for investigation. This gave us live, production-grade feedback on data divergence.
⁠
  \n  ⁠Batch Reconciliation\n
⁠	    •	Ran periodic batch jobs comparing large datasets.
	    •	Focused on high-value entities and critical tables.
	    •	Highlighted drift that wasn’t visible in real-time traffic. \nShadow reads caught immediate issues; batch scans caught structural ones.\n
⁠⁠Challenge #3: Handling False Positives \nNot every mismatch was a real bug. Common false positives included:\n    •	Timestamp format differences (UTC vs local time).
	•	Optional fields behaving differently.
	•	Legacy fields populated differently. Without careful handling, these harmless differences would swamp our metrics and slow real progress. 
⁠    
\n⁠What Worked:\n
	•	Accuracy workshops: Engineers, QA, and data stakeholders sat together, reviewed mismatch samples, classified real bugs vs benign differences.
	•	Refined comparison logic: Adjusted matching rules and tolerated known differences. 
⁠
⁠These workshops helped reduce noise and allowed us to focus on true blockers to 99.9% accuracy.\n
Lessons we took away\n    •	You can’t eliminate latency in event-driven systems. You design around it.
	•	Blind trust kills migrations. Metrics, visibility, and honesty build trust.
	•	Validation needs to happen at multiple layers. Shadow reads + batch reconciliations together made our system robust.
	•	Transparency speeds up trust-building. We shared dashboards and accuracy numbers — even when they weren’t great — with leadership and engineers alike.\n
⁠Conclusion\nTrust Was Built, Not Assumed At the beginning, our accuracy was zero. There were no shortcuts — only careful validation, transparency, and iteration. 
⁠By the time we moved forward, the new system had earned trust — with measurable proof behind it, not just good intentions. 

⁠In any critical migration, that’s the real success: Trust isn’t migrated automatically. It’s rebuilt — record by record, validation by validation, and through relentless honesty in every metric.

Moniepoint Engineering

Automating Kubernetes Infrastructure with Crossplane Functions

All Stories

Infrastructure, Research & thinking

Automating Kubernetes Infrastructure with Crossplane Functions

Infrastructure

Scheduled jobs in Java + Spring + Kubernetes - Part 2

Infrastructure

Scheduled jobs in Java + Spring + Kubernetes - Part 1

Infrastructure

What Running Kafka on VMs Taught Us About Systems Thinking

Culture

When Code Becomes Cheap, Thinking Becomes Expensive

Infrastructure

High Availability in Production: What Running at Scale Actually Requires

Automating Kubernetes Infrastructure with Crossplane Functions

Infrastructure, Research & thinking

Automating Kubernetes Infrastructure with Crossplane Functions

Infrastructure

Scheduled jobs in Java + Spring + Kubernetes - Part 2

Infrastructure

Scheduled jobs in Java + Spring + Kubernetes - Part 1

Infrastructure

What Running Kafka on VMs Taught Us About Systems Thinking

Culture

When Code Becomes Cheap, Thinking Becomes Expensive

Infrastructure

High Availability in Production: What Running at Scale Actually Requires

Get more stories like this