Most infrastructure stories get written after something breaks. This one is different. Nobody paged Celestina Amadi's team at 3 AM about Kafka. The VMs were working, yet she made the call to rebuild anyway because she could see where things were headed before they became a problem.

Celestina Amadi leads the Cloud Engineering team behind the infrastructure that powers Moniepoint's payment and savings products, building systems that move transaction and savings data reliably for millions of customers every day. She's also a Grafana Champion, a HashiCorp Ambassador, and an IBM Champion 2026.

Why We Moved to Strimzi

It Wasn't a Crisis. It Was a Decision.

We moved to Strimzi because we looked honestly at how we were running Kafka and made a deliberate call: this does not scale, and we can do better.

That kind of decision is actually harder than reacting to a crisis. Incidents are obvious — something breaks, and you fix it. Proactive architectural change requires you to see clearly enough to act on discomfort before it becomes a disaster. It requires you to say "this is working, but not well enough" and mean it.

This is the story of what we saw, what we built, and what it taught us about thinking in systems.

How We Were Running Kafka Before

Our Kafka setup started the way most things do in a fast-moving engineering team: pragmatically.

We needed CDC pipelines. Moving data from Database A to Kafka to Database B, from Database C to Kafka to Database D. Multiple pipelines, each serving a different business flow. We spun up Kafka instances on VMs, managed with Docker Compose as needed. They worked. We moved on.

Over time, the seams started showing.

• Every change required SSH and port-forwarding. Our Kafka instances had no URL. You accessed them via localhost. To make any change, whether adding a source connector, adding a sink, or updating a configuration, you had to SSH into the server, port-forward to get local access, and make the change manually. Every time. In every instance.

• With Docker Compose, downtime was always one command away. Managing Kafka with Docker Compose works until it doesn't. Any update or fix that required restarting the stack meant running `docker compose restart` and hoping the restart went cleanly. For a data pipeline, that's not a comfortable position to be in.

• Versions were going stale, and updates were per instance. Kafka evolves. New version released. But because each instance was a separate VM managed independently, upgrading meant going into each one individually. To make it worse, our instances weren't even consistent. Some ran on ZooKeeper, some had adopted Raft as Kafka evolved. Different consensus models, different operational behaviour, different things to know before touching anything.

• Provisioning a new pipeline meant reinventing the wheel. No standard process. No shared configuration. Every new CDC pipeline was a fresh set of decisions: which version, which consensus model, which configuration. Knowledge of how each cluster worked lived with whoever set it up.

• Monitoring was manual.  We used JMX Exporter to pull metrics and eventually shipped logs to Loki. These helped. But they were layers on top of an operational model that still required you to SSH into the right instance to actually diagnose anything. There was no unified view of the fleet.

None of this was broken in a way that triggered an alarm. It was just slow, manual, and increasingly inconsistent as the fleet grew. We were spending engineering time on operations that should have been automatic.

What Is Strimzi?

Before getting into how we implemented it, it helps to understand the core idea.

Strimzi is Kafka running on a Kubernetes cluster, managed by a Kubernetes operator. An operator is a piece of software that runs inside your cluster and continuously manages another application on your behalf. It watches a set of configuration files and ensures the actual state of your Kafka deployment always matches what you declared. If something drifts, the operator corrects it. If a broker goes down, the operator brings it back.

The key shift: you stop being the operator. The software becomes the operator.

Strimzi is open-source and part of the CNCF — no licensing costs, no vendor lock-in. Everything is expressed through Kubernetes custom resources (CRDs), which means your entire Kafka setup is code that lives in Git.

How We Implemented It

We didn't just drop Strimzi onto an existing cluster and call it done. We made deliberate decisions about how to structure it.

Dedicated Kubernetes cluster: We created a dedicated cluster for our Kafka workloads on GCP/GKE. Separating Kafka onto its own cluster gave us clean resource boundaries and made it easier to manage capacity independently from our application workloads.

Namespace isolation per pipeline: Within that cluster, each Kafka deployment lives in its own namespace. The pipeline from DB A to DB B gets its own namespace. DB C to DB D gets another. Instead of "one VM per pipeline," we now have "one namespace per pipeline" — with real isolation between them. A misconfiguration in one namespace cannot consume resources or affect availability in another.

YAML templates as the standard: We created standardised Strimzi cluster templates. When a new pipeline needs a Kafka instance, you take the template, fill in the relevant fields, and apply it. Each template covers the full stack: the Kafka cluster definition, Kafka Connect, the node pool, and metrics configuration. Every new instance comes out consistently because it comes from the same starting point.

ArgoCD for sources, sinks, and configuration: Every connector (source, sink, configuration) is defined in a YAML file and lives in a Git repository. To add a new source or sink, you write a YAML file and open a merge request. Once it's merged, ArgoCD picks up the change and automatically syncs it to the cluster. The operator applies it from there.

Custom Kafka Connect image as a single source of truth: We extend the official Strimzi Kafka image and bake in all the connectors we use. That includes Debezium connectors for CDC (PostgreSQL, MySQL, and Spanner) plus ClickHouse, MongoDB, S3, and the Confluent JDBC connector with the relevant JDBC drivers. All connector versions are pinned explicitly in the Dockerfile. When a connector needs an upgrade, we update a single line in the Dockerfile and build a new image. When a new Kafka version is released, we rebase onto the updated Strimzi image. One place. One PR. The operator rolls it out across all clusters.

What Changed Day-to-Day

The difference in day-to-day operations was immediate.

No more SSH: No more port-forwarding. To add a source or sink, you write YAML and push. The operator applies it. What used to require terminal access to a VM is now a pull request that gets reviewed and merged.

No more Docker Compose restarts: Strimzi manages the broker lifecycle. You don't `docker compose down` a production data pipeline anymore.

Scaling is a number in a file: Need an additional broker? Change one value in your YAML. The operator handles placement and ensures the cluster state stays consistent. Need to scale down? Same. It handles graceful decommissioning.

Logging without the archaeology: Every broker's logs are available via standard Kubernetes tooling — `kubectl logs` from the terminal, or directly in Lens and k9s if you prefer a UI. We still ship to Loki for retention and cross-cluster querying.

A single UI that shows everything at once: We use Kafbat internally, which gives us a unified view across all clusters, including broker health, connector status, consumer lag, and topic metadata. When a connector fails, you can see why it failed, inspect the configuration, and understand the issue from a browser. No terminal required.

Rolling updates without downtime: Kafka upgrades happen one broker at a time, with the operator ensuring quorum is maintained throughout. Most of our cluster updates now happen without downstream systems noticing.

Self-healing: If a broker crashes, the operator restarts it. If Kubernetes can't schedule it on the current node, it reschedules elsewhere. You get an alert that something happened and was resolved — not a page asking you to fix it manually at an inconvenient hour.

A Few More Reasons Worth Naming

The narrative above covers most of it, but three points don't show up in the day-to-day story.

• Cost efficiency: Strimzi is open-source with no licensing fees. Running Kafka inside the same Kubernetes cluster as your consumers and producers also cuts network egress costs you'd otherwise pay for traffic crossing VPC boundaries to reach an external Kafka endpoint.

• Lower latency: When producers and consumers live in the same cluster as Kafka, the network path between them is short. External setups add hops. For high-throughput CDC pipelines, that difference is measurable.

• Environmental parity: Because your entire setup is YAML in Git and synced by ArgoCD, cloning it from production to staging is straightforward — fill in a template, point it at a different namespace, and apply. The same configuration that runs in production can run locally in Minikube for debugging.

This was the right call for us, but it's not universal.

Strimzi is worth it if:

• You're already running Kubernetes, or seriously committed to it
• You're managing multiple Kafka pipelines with inconsistent configurations
• Your team is spending real time on SSH access, manual connector management, or per-instance version updates
• You want GitOps as a genuine practice — not just a concept
• You have the Kubernetes maturity to manage operators safely

Strimzi probably isn't the right fit if:

• Kafka is peripheral to your product. Use managed Confluent Cloud or MSK instead
• Your organisation isn't comfortable with Kubernetes yet
• You have a single, simple, well-tuned cluster that isn't causing problems
• You need commercial support SLAs. Strimzi is community-backed, not commercially supported
• You want pre-built integrations for monitoring, alerting, or schema registries out of the box. With Strimzi, you wire those up yourself

For us, the signals were clear: multiple pipelines running inconsistently, SSH and Docker Compose eating engineering time, no unified visibility, and an existing Kubernetes investment. The only real question was why we hadn't done it sooner.

The Lesson

You can't think in systems when you're drowning in operations.
Looking back, the biggest thing running Kafka on VMs taught us wasn't about Kafka. It was about the manual infrastructure costs that you can't easily measure.

Each individual task was manageable. But collectively, the manual work consumed the cognitive space that should have been available for thinking about the system as a whole. We were so busy managing the current state of things that we never stepped back to design how they should work.

Moving to Strimzi didn't just reduce our operational burden; it changed what our team was able to think about.

When the infrastructure is code, you can reason about it. When operations are declarative, you can predict them. When complexity is visible in Kafbat instead of buried across disconnected instances, you can make architecture decisions rather than just firefighting.

New engineers can read the YAML and understand the entire Kafka setup. Changes are reviewed before they happen. The system tells you when something goes wrong and often fixes itself before you get involved.

That's the shift that matters: from operators who keep things running to engineers who design systems that run themselves.

Decisions like the one in this article don't come from chasing incidents. They come from a team with the standards to ask "is this actually good enough?" before something forces the question.

If you'd love to build systems that run themselves rather than spend your career keeping systems from falling over, we're looking for you.

Explore open engineering roles at Moniepoint.