Moniepoint welcomed Dr Victor Odumuyiwa, Research Lead, Machine Intelligence Research Group, UNILAG.\nIn his presentation, he explored how transformer models are applied to financial texts to uncover customer insights.
⁠
⁠
⁠\n\n
⁠Objective and Introduction\nThe fundamental challenge in leveraging machine intelligence is \nsemantics\n. How do we ensure machines understand not just the text, but the context surrounding that information? A vast array of possibilities opens up if a machine can truly understand human communication.\n\nFor organisations dealing with high volumes of customer interactions, this context is crucial. Natural Language Processing (NLP), a subfield of Artificial Intelligence (AI), focuses on programming computers to process and analyse human language data. This guide will detail how Transformer models, the current state-of-the-art in NLP, can be applied to financial texts to generate actionable insights automatically.\n
⁠Prerequisites / What You'll Need\nFamiliarity with foundational AI/ML concepts (RNN, LSTM, Neural Networks).\nBasic understanding of NLP tasks (Classification, Translation, Generation).\nKnowledge of metrics used for model performance evaluation (Precision, Recall/Sensitivity, Accuracy).\n
⁠The Foundation: The Rise of the Transformer Architecture\nThe journey toward contextual understanding evolved through several stages: starting with Recurrent Neural Networks (RNN), moving to Long Short-Term Memory (LSTM), and culminating in the \nTransformer\n architecture.
\nA key limitation of earlier models like LSTM was the difficulty in maintaining context over long sequences (the vanishing gradient problem), even with improvements like Bi-LSTM (processing in both directions).\n\nThe breakthrough came in 2017 when Google published the paper, \n"Attention is all you need"\n. Attention mechanisms changed the flow of NLP by enabling the model to remember and connect words to past words, understanding relationships across the entire sequence, which is critical for human language comprehension.\n
⁠Core Components of the Transformer\nThe Transformer relies on several mechanisms to capture semantic meaning and positional context:\nTokenisation and Embedding:\n Input text is broken down into numerical tokens (words or subwords), which are then converted into numerical vectors (embeddings) to capture their semantic meaning.\nPositional Encoding:\n Since the model processes input in parallel, vectors are added to the input embeddings to provide information about each token's position, ensuring the model understands the order of the sequence.\nSelf-Attention Mechanism:\n This is the core component that determines the relationship between all tokens in a sequence simultaneously, using three vectors for each token:\nQuery (Q): Represents the current token being processed.\nKey (K): Acts as a label for every other token in the sequence.\nValue (V): Contains the actual content of the tokens. By computing attention scores between the Query and all Keys, the model learns how much attention to pay to each Value, forming a weighted, contextual representation.\nBERT (Bidirectional Encoder Representations from Transformers):\n Introduced in 2018, \nBERT\n further refined this by processing text in more directions, leading to better contextual representations. This model started the trend of predicting the next word using masked input, enabling effective content generation.\n
⁠Key NLP Tasks in Financial Services\nThese advanced models are applied across several critical NLP tasks, moving far beyond simple text processing to semantic understanding.\n
⁠Sentiment Analysis (Customer Opinion)\nSentiment analysis determines the opinion expressed in a piece of text (e.g., positive, negative, neutral, angry, or hateful). This is essential for understanding customer intimacy and public perception, especially regarding new products or services.
\nIn testing various models on financial datasets (including customer complaints and financial news), \nBERT\n consistently showed better \nAccuracy, Precision\n, and \nRecall\n results compared to traditional machine learning models like SVM (using TF-IDF) and older neural networks like LSTM. For example, on one financial sentiment analysis dataset (Dataset 2), BERT achieved 83% accuracy, significantly outperforming SVM (68%) and LSTM (70%).\n
⁠Financial Named Entity Recognition (NER)\nNER automatically identifies a document's names and specific entities (like countries, people, or organisations). This capability is critical for relationship extraction and synthesising information from millions of business news articles and documents, giving financial professionals a concise summary and insight.\n\nNER models, using optimisers like AdamW and loss functions like Cross-Entropy, can accurately tag specific entities in sentences, whether using BERT or Llama models.\n
⁠Machine Translation\nSignificant progress has made machine translation highly effective today, unlike the difficulties encountered years ago. Systems leveraging Transformer architectures (like Google's and OpenAI's models) can now translate complex financial documents, such as news concerning an $800 million World Bank grant to the Nigerian government, moving beyond literal translations to retain the general literary meaning.\n
⁠Financial Question Answering (QA)\nFinancial QA involves training a model on specific data, such as company financial reports (10-K filings). The trained machine can then accurately respond to particular questions, reducing the need for humans to handle routine queries. Models like \nRoBERTa\n have been trained to answer questions regarding financial details, such as the value of a one-time transaction tax.\n
⁠Generating Actionable Customer Insights\nThe true value of these NLP tasks lies in combining them to generate comprehensive customer insights, rather than treating them as scattered tasks.\nA system can take unstructured data (like a customer complaint), perform several tasks such as \ncategory detection\n (e.g., Debt collection), \npriority detection\n (e.g., Urgent), and \nsentiment detection\n (e.g., neutral) and then plug this classified information into an LLM or Transformer to generate detailed summaries.\nExample Insight Generation from a Customer Complaint: Based on a complaint detailing an unrecognised debt, an unauthorised account levy, and communication failures due to an outdated address, the combined system generates multi-faceted insights:\nDisputed Debt Handling Issues:\n The required process (affidavit of fraud, police report) was overly burdensome, leading to unresolved disputes.\nCommunication and Notification Failures:\n Legal correspondence was sent to an outdated address, despite the company having the correct, current address on file from a recent payment.\nData Management Concerns:\n This situation reveals a clear breakdown in maintaining and cross-referencing up-to-date customer information, suggesting poor data management.\nCustomer Impact:\n The resulting judgment caused a financial loss ($830 levied) and significant emotional distress for the customer.\nFinally, the system suggests \nProcess Improvement Opportunities\n and \nActionable Insights\n, such as implementing robust address verification protocols before legal actions and simplifying the debt dispute process.\n
⁠Troubleshooting & FAQ\nQuestion:\n When is it most efficient to fine-tune an LLM versus prompt engineering?\nAnswer:\n Fine-tuning and prompt engineering serve different goals, but both depend heavily on the context.\nFine-tuning\n is primarily used to adapt a general model (pre-trained on broad language) to a \nspecific domain\n or form (e.g., financial jargon, JSON output format). You are modifying the model’s parameters to understand your particular data better. For fine-tuning to be effective, you need a substantial dataset—hundreds of thousands of examples, as smaller datasets may yield inaccurate results.\nPrompt Engineering\n (or prompt tuning) helps provide specific \nfacts\n or dictate behaviour, but the size of the context window limits it. If your context is extremely large (e.g., many data sources), prompt tuning may not be viable due to token limits.
\nQuestion:\n For Financial Question Answering, should we use Retrieval-Augmented Generation (RAG) or train from scratch?\nAnswer:\n While you can train a small language model entirely from scratch if you have sufficient data and compute power, RAG is generally recommended for integrating real-time or proprietary financial data.\n
\nRAG augments the knowledge of a pre-trained model (which is limited by the information available during its training, perhaps six months ago) by fetching current, external, or private information. This is necessary to integrate new data into the model’s knowledge base without retraining the expensive base model every day. RAG also provides a mechanism to allow an LLM to interact with sensitive resources, like your internal database, by creating a system that extracts context-specific information for generation, while keeping the database secure.\n
⁠Conclusion: The Road Ahead for African AI\nTransformer models provide unprecedented power for extracting insight from text. However, the future of AI in regions like Africa requires a focus on AI localism.
\nSince most African languages are spoken rather than written, focusing on multimodal data (spoken and acoustic data) is essential to drive financial inclusion. Currently, most African languages (like Yorùbá, Igbo, and Hausa) are poorly represented in global AI tools due to factors like the absence of standardised orthographies and the paucity or absence of digital corpora (web-scale data).\n\nTo maximise the benefits of AI for African populations, there must be intentionality and strong collaboration between industry, academia, and philanthropy. The strategic direction should focus on building and using Small Language Models (SLMs) tailored to local contexts, rather than trying to compete in the race for Artificial General Intelligence (AGI). This focus will ensure that ordinary people, such as small business owners, can benefit from AI by receiving truly contextual and relevant advice.\n

Unlocking Customer Insight: Applying Transformer Models to Unstructured Financial Text

Moniepoint welcomed Comfort Lawal, Assistant Lecturer, Covenant University.
⁠
Presenting a cutting-edge approach to anomaly detection in distributed cloud systems. Using Graph Neural Networks (GNNs), this method facilitates the identification of unusual patterns in log data, thereby helping cloud environments remain secure, stable, and reliable.

⁠\n\n
⁠Introduction\nIn the modern digital economy, complexity is the new normal. Enterprises rely on multi-cloud architectures (used by 73% of organisations) that generate an astounding 500+ billion logs daily. This massive, distributed infrastructure presents a severe challenge for security and reliability, especially considering that the global annual cost of cybercrime has reached €1.75 trillion. Detecting critical system failures or malicious activity requires an immediate response, often less than 1 second.
⁠\nTraditional anomaly detection methods, such as centralised monitoring and isolated rule-based systems, fail to meet this challenge. They often suffer from privacy violations, bandwidth bottlenecks, high false positive rates, and an inability to adapt to new threats.\n
⁠The Problem with Traditional Approaches\nTraditional security and monitoring methods fall short in these distributed, high-volume environments.\nCentralised Monitoring:\n Fails due to privacy violations when raw data is pooled, creating single points of failure and causing bandwidth bottlenecks.\nIsolated Detection (Classical ML):\n Treats data as independent, leading to high false positive rates. For instance, a poor network connection might be falsely flagged as an anomaly because the system fails to check the related data context.\nRule-Based Systems:\n Cannot adapt to new or evolving threats and require constant, manual maintenance.\n\nThe Solution:\n Converging Technologies\nThe Federated Graph Neural Network (F-GNN) framework is an innovation designed to overcome these critical limitations by combining three powerful technologies:\nGraph Neural Networks (GNNs):\n Capture complex relationships in vast network data, which is a natural fit for cloud systems.\nFederated Learning (FL):\n Allows models to be trained without sharing raw, sensitive data, enabling privacy-preserving collaboration across distributed sites.\nHyperbolic Geometry (HGNN):\n Provides an efficient way to model the hierarchical nature of cloud architectures.\n
⁠Prerequisites / What You'll Need\nDeploying and experimenting with F-GNNs requires expertise and infrastructure across several domains:\nExpertise:\nMachine Learning Engineers with GNN expertise.\nSecurity Engineers experienced in privacy protocols.\nDevOps Engineers for infrastructure management.\nCore Frameworks:\nFlower: Recommended framework for Federated Learning.\nPyTorch Geometric (pyg.org): Used for GNN implementation.\nDGL (dgl.ai): Crucial for distributed graph processing, especially when dealing with large graphs (e.g., 66,000+ nodes and millions of edges).\n
⁠Core Concepts and Technical Deep Dive\nFoundational Concepts:\n Understanding the Building Blocks\nGraph Neural Networks (GNNs)\nCloud systems are a natural fit for graph representation. In this context:\nNodes represent components like Servers, VMs, or Containers.\nEdges represent relationships such as Network connections, Dependencies, or API Calls.\nFeatures include metrics, logs, and system states.\nThe key advantage of GNNs is Message Passing, where nodes learn from their neighbourhood to reduce false positive rates and solve the challenge of isolated detection by considering underlying relationships.\n
⁠Common GNN architectures include:\nGraph Convolutional Networks (GCN):\n Fast and efficient, suitable for static graphs, but perform transductive learning, requiring retraining for new data.\nGraph Attention Networks (GAT):\n Uses an attention mechanism for dynamic importance, often used for threat detection, but it is of higher complexity.\nGraphSAGE:\n Supports inductive learning through scalable sampling, making it suitable for large and constantly changing networks.\n
⁠The Hyperbolic Advantage\nWhen dealing with deeply hierarchical data—such as the structure of a cloud environment (Data centre → Cluster → Node hierarchies)—Euclidean space introduces problems like poor hierarchy preservation and high dimensions. Hyperbolic Geometry naturally caters to these hierarchies, offering:\nNatural hierarchy embedding.\nEfficiency: Requiring 10x fewer parameters.\nPerformance Gain: Demonstrated 16-63% performance gains in the literature.\n
⁠Federated Learning (FL)\nFL allows the training of a Global Model continuously improved by training on decentralised, local graph data residing on different cloud sites that form a federation. This process involves:\nTraining models without sharing raw data.\nEnsuring privacy-preserving collaboration.\n
⁠The Innovation: The Federated GNN Architecture\nThe F-GNN framework operates via a continuous learning cycle coordinated across a decentralised network.
\n1. Local GNN Training\nEach participating cloud centre or edge server trains a GNN model locally using its site-specific data and logs. This layer ensures that raw data privacy is preserved. The local GNN calculates an anomaly score for each node.
⁠\n2. Secure Aggregation\nAfter local training, only the model updates (gradients) are sent to the global coordination server, not the raw data. These updates are protected using advanced privacy techniques:\nPrivacy Technique\nFunction/Guarantee\nDifferential Privacy\nAdds noise injection to the model updates to mask specific information, providing mathematical privacy guarantees\nHomomorphic Encryption\nAllows computation on encrypted data (full privacy) and can involve quantum-resistant options\nSecure Aggregation\nUses secret sharing protocols and Byzantine fault tolerance to protect the combined model updates\nCrucially, implementing security involves a performance vs. privacy trade-off. Testing showed that while full encryption (Homomorphic Encryption) achieved high privacy, accuracy dropped to 85%. Using Differential Privacy (medium privacy) yielded an accuracy of approximately 97%.\n
\n3. Global Model Distribution\nThe model updates are securely aggregated (often using the FedAvg algorithm with enhancements like gradient clipping) to create an improved global model. This global model is then distributed back to all local sites, completing the continuous learning cycle. The global coordination server can sit in any one of the federating cloud providers.\nMulti-Layer Privacy Protection\nTo secure the model updates transmitted between local clients and the global server, the framework employs multi-layer protection techniques:\nDifferential Privacy (DP):\n Provides mathematical privacy guarantees by adding noise injection to the model updates, masking the information.\nHomomorphic Encryption (HE):\n Allows computation on encrypted data, offering full privacy and quantum-resistant options, although it can significantly impact model accuracy.\nSecure Aggregation:\n Uses secret sharing protocols and Byzantine fault tolerance to ensure secure computation on the updates.\nPerformance Breakthroughs\nImplementing these security measures must not negate the necessity for real-time response. The FGNN framework achieves groundbreaking efficiency through several optimisation techniques:\nCommunication Optimisation:\n The system employs Asynchronous Updates (where clouds send updates based on customised rules, like volume or accuracy changes), Top-k sparsification, and Gradient compression. This optimisation, achieved through methods like FEDZIP, resulted in a 97.6% bandwidth reduction. Communication time dropped from 700 seconds per round to 16.8 seconds per round.\nModel Compression:\n Techniques like Quantisation (reducing updates from 32-bit to 8-bit without information loss), Pruning (removing unnecessary parameters, achieving 50% removal), and Knowledge Distillation (where a faster student model learns from a robust teacher model) provide a 10x speedup and ensure production readiness.\n\nReal-World Impact: Results That Matter\nThe FGNN framework provides high Privacy+Performance.\nMetric\nRange/Value\nInsight\nDetection Rate\n85.02% - 99.998%\nAccuracy for DoS attacks is near-perfect (99.998%).\nLatency\n< 100ms\nEnables true real-time critical alerts.\nCommunication Reduction\n97.6%\nBreakthrough in bandwidth optimisation.\nEnergy Preservation\n99%\nSignificant efficiency gains.\nPrivacy Cost\n12% loss in accuracy\nFull encryption (High Privacy) reduces accuracy from 99.6% (No Protection) to 87.85%.\n\nKey deployment sectors include:\nFinancial Services:\n Used for cross-institution fraud detection, maintaining 99.2% accuracy with zero data breaches.\nCloud Providers:\n Applicable for cost anomaly detection (AWS), cognitive service integration (Azure), and edge deployment readiness (GCP).\nHealthcare:\n Enables distributed patient monitoring and HIPAA-compliant, cross-hospital learning.\nTroubleshooting & FAQ\nQuestion:\n Traditional models often suffer from poor explainability. How does this deep learning approach provide sufficient justification for an anomaly?\nAnswer:\n Explainability is primarily achieved through hierarchical classification, which goes far beyond a simple "anomaly/not anomaly" output. The system breaks down the alert into different levels:\nDetection: An anomaly occurred.\nType of Anomaly: Is it a critical anomaly (system about to fail), an error anomaly, or a warning anomaly (requires different actions)?\nApplication: Which cloud application produced the anomaly (e.g., computing, messaging, networking)?\nSpecific Model/Component: Pinpoints the exact component causing the issue, allowing administrators to target their troubleshooting efforts.\nThis hierarchical breakdown makes the system more transparent, although it currently stops short of explaining the ultimate "why" behind the specific component failure.\n
\nQuestion:\n Are there standard comparative works that show whether Homomorphic Encryption or Differential Privacy is definitively "better" for F-GNN implementation?\nAnswer:\n To the best of current knowledge, a standard work comprehensively comparing all privacy or optimisation algorithms in a federated graph learning context does not yet exist. However, practical experimentation revealed a significant trade-off: Homomorphic Encryption (offering complete privacy) caused the accuracy to drop to 85%, while Differential Privacy (medium privacy) maintained roughly 97% accuracy. Choosing the correct protocol depends on whether the organisation prioritises maximum privacy or performance.
\nQuestion:\n When comparing F-GNN to centralised or classical models for anomaly detection, how much better is the federated graph approach?\nAnswer:\n Comparative analysis showed that the F-GNN model provided a 15.7% improvement in accuracy compared to existing anomaly detection works, including centralised and flat classification approaches. Furthermore, the F-GNN model is built with the intelligence to automatically tweak its parameters (like learning rates) based on local data characteristics. It is crucial for solving convergence challenges associated with non-Independent and Identically Distributed (non-IID) data in federated environments.\nConclusion\nThe Federated Graph Neural Network framework represents a pivotal step forward, enabling organisations to address massive data volumes and critical, sub-second latency requirements while strictly adhering to privacy protocols.
\nBy leveraging GNNs, Federated Learning, and the efficiency of Hyperbolic Geometry, this approach achieves 99%+ accuracy while maintaining complete privacy compliance. The breakthrough communication efficiency, reaching over 97% communication reduction, makes true, real-time federated learning viable.\n
⁠Next Steps and Future Vision\nThe journey toward fully optimised, secure AI is ongoing. Organisations and practitioners should:\nInvest in Privacy-Preserving Technology:\n Prioritise technology that maintains regulatory compliance while offering a competitive advantage through collaboration.\nExplore Novel Architectures:\n Researchers are encouraged to explore advanced areas like hyperbolic GNN variants and hybrid approaches (e.g., Transformer-GNN fusion) to further improve detection rates, especially since theoretical work suggests HGNN can dramatically simplify hierarchical classification.\nFuture Integrations:\n The framework is built to be future-proof, with planned integration for Large Language Models (for natural language anomaly explanations) and Quantum Computing (for ultra-low latency processing).\n

Federated Graph Neural Networks: Solving Real-Time Anomaly Detection Without Sacrificing Privacy

Moniepoint \nwelcomed \nErnest E, Onuiri, Ph.D, \nTo share his knowledge on precision medicine, and he’s learning from machine learning for cancer detection. He shared a foot on the round view of what is happening in academic research.\n
⁠⁠Introduction\n
⁠The promise of healthcare often feels like a distant ideal, especially when faced with the "one-size-fits-all" approach to treatment. Imagine receiving medicine for malaria that works for some, but not for you, or a cancer treatment that cures one patient but harms another. This disparity highlights a fundamental problem that precision medicine aims to solve: tailoring medical interventions to the individual. But while the vision of customised care is compelling, its widespread adoption faces significant challenges, particularly around scalability, data complexity, and a collective reluctance to embrace change. We will explore the core concepts of precision medicine, delve into the obstacles hindering its scalability, and propose pathways for collaboration between industry and academia to build the trustworthy AI systems needed to revolutionise healthcare.\n
⁠What is Precision Medicine?\nAt its heart, precision medicine is about moving beyond generic treatments to customise care for each individual. The goal is to minimise adverse drug reactions and autoimmune problems that can sometimes be more harmful than the disease itself. This personalised approach relies on a rich collection of data points, including:\nClinical Records:\n Observable features like temperature, weight, and responses to basic questions gathered during a hospital visit. These are generally easy to understand.\nGenomic Landscape:\n The complex internal data within humans often explains why things go wrong internally before external manifestations appear. This data is not easily accessible, expensive to obtain, and requires specialised tools and expertise to interpret, often appearing like noise to the untrained eye.\nImaging Data:\n Various forms of visual data obtained through technologies like MRI, CT, ultrasound, or X-ray.\nSeveral key principles underpin precision medicine:\nPersonalisation:\n Customising medicine for a patient based on their specific needs.\nPrediction:\n The ability to anticipate health events, which enables proactive prevention and treatment.\nParticipation:\n Empowering the patient to participate actively in their healthcare journey, asking questions, understanding the proposed treatments, and giving a sense of ownership.\n
⁠The Scalability Gap: Challenges in Precision Medicine\nDespite its promise, precision medicine encounters substantial hurdles in scaling beyond controlled research environments to widespread clinical adoption.\nData Complexity and Heterogeneity:\n Diseases like cancer are inherently heterogeneous, with various subtypes and severity levels. Genomic data, unlike simple clinical records, is incredibly complex and requires intermediate tools and programming languages (like R) to be unpacked and made intelligible. Combining different data modes (clinical, genomic, imaging) further exacerbates this complexity.\nData Integration and Quality:\n Healthcare data is often siloed, stored in disparate locations (e.g., Lagos, London, Tokyo) and in formats that appeal to local practices, making integration a hurdle for researchers. Furthermore, data is rarely clean, often containing missing values, noise, or being jumbled, leading to wasted research time and unreliable insights. Even long-standing repositories in institutions may not be trustworthy for new applications because the initial workflows were not designed for broad utility.\nCost and Accessibility:\n While the technology exists for genomic sequencing, it remains prohibitively expensive and is not readily available in most hospitals, limiting its widespread application.\nReluctance to Change:\n A significant challenge is the human element – the resistance to adopting new, unconventional approaches. Experienced medical professionals may adhere to traditional methods, viewing new technologies like Electronic Medical Records (EMRs) with suspicion, fearing activity tracking rather than appreciating their potential for record-keeping and knowledge transfer.\nLack of Collaboration:\n There's often a disconnect between academia and industry. Academia might pursue theoretical solutions, while the industry focuses on immediate monetizable products, which are missing opportunities for collaboration. Academic studies, especially in certain parts of the world, often lack the funding and time to generate their own clean, fit-for-purpose data, forcing reliance on external sources.\n
⁠Lessons from COVID-19 and the Importance of Validation\nThe COVID-19 pandemic offered crucial lessons, particularly regarding the need for rigorous validation and the potential of time compression in drug discovery. The rapid development of vaccines, facilitated by in silico (computational) methods, showcased how processes that once took years could be dramatically accelerated. However, this unprecedented speed also led to questions about the longevity and nature of these interventions, as evidenced by the need for multiple doses and boosters, unlike traditional vaccines.\nThis experience underscores that validation is a never-ending process. Engineers and researchers must cultivate a "inquisitive mind" that constantly asks "what if," even when a solution works perfectly. The speaker also emphasises that simplicity often trumps complexity in engineering solutions. While complex models seem more robust, simpler approaches can usually yield equally good, if not better, outcomes with less computational burden and more efficient resource utilisation. Over-complicating solutions just for the sake of it can lead to systems struggling and ultimately failing to deliver effectively.\n
⁠Pathways to Bridging the Gap: The Power of Collaboration\nOvercoming precision medicine's scalability gap demands a concerted effort and a fundamental shift towards synergistic collaboration between industry, academia, and regulatory bodies.\nIndustry-Academia Alignment:\n When industry (with its resources and real-world problems) and academia (with its research and innovative thinking) combine forces, they create something far greater than they could achieve alone. This symbiosis can translate research into monetizable solutions that address real industry needs, as seen in places like Silicon Valley's relationship with Stanford University.\nShared Data Repositories:\n Industry players, especially those generating vast amounts of data, should partner with academia to create shared data repositories. This would allow researchers to use relevant, homegrown data, leading to more tailored findings and solutions directly applicable to local contexts, rather than constantly adapting imported data and solutions.\nCustomisation in Education:\n Education itself needs to be customised to solve real problems, encouraging students to appreciate the practical application of their learning.\nLeveraging Technology for Collaborative Workflows:
\nModern technology has eliminated geographical barriers, enabling remote collaboration. This capability should be fully leveraged to foster interdisciplinary teams that bring diverse perspectives and expertise to complex problems, preventing work in silos.\n
⁠Trustworthy AI: Shared Principles Across Healthcare and Fintech\nThe need for robust, explainable, and scalable AI systems is not unique to healthcare. Industries like fintech face remarkably similar challenges and benefit from similar principles.\nRisk Prediction:\n As healthcare professionals worry about clinical risk prediction, fintech deals with financial risk prediction.\nPersonalisation:\n Precision medicine aims for personalised patient treatment; fintech strives for personalised customer recommendations.\nClassification:\n Disease subtype classification in medicine finds its parallel in customer segmentation and classification in fintech.\nFraud and Anomaly Detection:\n Protecting sensitive patient data and detecting anomalies in healthcare is mirrored by the need for financial fraud and anomaly detection, ensuring data protection and compliance.\nIn both sectors, trustworthy, explainable, and scalable systems are vital for adoption. The computational complexity often seen in precision medicine, which is a tall order, also presents a significant opportunity for innovation and scaling.\n
⁠
⁠Technology: Ability to change data into medicine\nDr. Onuiri's approach to precision medicine involves a structured workflow to transform complex, multimodal data into actionable clinical insights. Building a trustworthy learning system requires more than a robust algorithm; it demands a rigorous, multi-step process from data collection to validation.
⁠
⁠\n\n
⁠Step 1: Multimodal Data Integration and Preprocessing\nThe first major hurdle is aggregating and harmonising disparate data types into a cohesive dataset.\nExplanation:\n Data for a single patient often exists in different formats and locations, such as clinical records from a hospital's EMR, genomic data from a specialised lab, and imaging data from a radiology department. These data silos and varying standards make integration a significant challenge.\nInstruction:\n The initial step is to build a comprehensive patient profile. This involves addressing challenges like \ndata integration\n, handling \nmissing values\n, and cleaning \nnoisy data\n to ensure the quality and interoperability needed for an effective model.\n
⁠Step 2: Feature Engineering\nWith an integrated dataset, the next step is to extract meaningful signals that a machine learning model can interpret.\nExplanation:\n Raw genomic or imaging data is often too complex and high-dimensional for a model to use directly. This critical step translates the data into a structured format, identifying key \nfeatures\n or patterns—like specific gene expression levels or image textures—that correlate with particular health outcomes.\nInstruction:\n Apply domain-specific techniques to convert the raw, multimodal data into numerical or categorical features. For example, specific gene expression levels might be used as features for a cancer classification model, while texture analysis could extract features from an MRI scan.\n
⁠Step 3: Model Development and Selection\nThis is where the "learning" happens. The goal is to choose and train a model that can perform a specific task, such as predicting clinical risk or classifying disease subtypes.\nExplanation:\n While it can be tempting to use the most complex model available, Dr. Onuiri emphasises that simpler, more interpretable models often yield equally good, if not better, outcomes with less computational burden. The focus should be on creating an accurate, efficient, and explainable model.\nInstruction:\n Train various models and evaluate them based on a balance of performance, computational cost, and \nexplainability\n. In a clinical setting, a transparent model clinicians can understand and trust is often preferable to a "black box" algorithm.\n
⁠Step 4: Rigorous and Continuous Validation\nA model is only valid if it is trustworthy. Validation is not a one-time check before deployment but a continuous, iterative process.\nExplanation:\n The COVID-19 pandemic provided a critical lesson in the importance of rigorous validation. A model that performs flawlessly on a historical training dataset might fail when exposed to new, real-world clinical scenarios. Engineers and data scientists must maintain an "inquisitive mind," constantly questioning the model's performance and limitations.\nInstruction:\n Continuously test the model against new data in different clinical contexts. This ongoing validation is crucial for ensuring the system remains \nreliable, safe, and effective\n over time, forming a trustworthy AI system's bedrock.\n
⁠Conclusion\nPrecision medicine holds immense potential to transform healthcare from a generalised approach to a deeply personal and effective one. However, this future hinges on our ability to navigate the complex data landscape, overcome resistance to change, and foster unprecedented collaboration. By promoting a symbiotic relationship between industry, academia, and regulatory bodies, we can collectively plug the scalability gap, build trustworthy AI systems, and unlock the breakthroughs that have long eluded us, leading to better outcomes for everyone.\n

Designing trustworthy learning systems in precision medicine and the scalability gap

\n
Introduction\nIn this article, I share my experience contributing to the migration of a critical system from a live monolith to a new microservice, focusing on the challenges of ensuring 99.9% data accuracy before full cutover. I cover how we handled eventual consistency, validated data across two systems, and built trust through rigorous engineering practices.\nMigrating critical systems isn’t just about launching new services.
⁠\n
It’s about migrating trust. Not long ago, I was part of a team working on a major system migration: moving important business logic from a live, production monolith to a brand-new microservice with its own Postgres database. The old monolith stayed live, serving 100% of public traffic.
⁠
The new system had to prove itself — from zero — before anyone could rely on it. Our mission was clear: \nProve 99.9% data accuracy before any cutover. 
⁠\n
When we started, our accuracy was effectively 0%. This is a look back at how we approached that challenge: the strategies, the technical lessons, and how real trust is built — slowly, carefully, and with metrics.\n
⁠Setting the Stage: Monolith to Microservice - The migration involved\n•	Old System: \nA mature monolith handling millions of transactions.\n
•	New System: \nAn event-driven microservice architecture backed by Postgres.\n
•	Kafka: \nStreaming live updates and historical backfills to the new system.\n
•	Grafana + Prometheus: \nInstrumentation and monitoring for everything.\n
•	Traffic: \nStill fully routed to the monolith during migration. \n
⁠
⁠This wasn’t a lift-and-shift. It was a rebuild — while the airplane was flying.\n
⁠⁠Why Accuracy Started at Zero Syncing? \nTwo complex systems asynchronously means you start from a simple engineering truth:\nNothing is accurate until you can prove it. At the beginning:
	•	No full validation pipeline.
	•	No accuracy metrics.
	•	No live comparison in place. Without verification, assuming correctness would have been reckless.
⁠
We treated data accuracy as seriously as uptime — building it into our operational SLOs.\n
⁠⁠How We Measured Data Accuracy?\n⁠We immediately built the core metrics:\nrecords_processed_total.\nrecords_mismatch_total.\nAccuracy Rate = (processed - mismatches) / processed. \nThese were exposed through Prometheus and visualized in Grafana dashboards:\nAccuracy rate trends.\nMismatch breakdowns.\nKafka consumer lags.\nError rates across different domains. \nWe also split comparisons into:\nFresh comparisons (new events, possibly still settling).\nAged comparisons (data old enough to have converged). 
\nThis distinction kept our metrics honest and alerts meaningful.\n
⁠Challenge #1: Latency and Eventual Consistency\nBecause the new system was event-driven, eventual consistency was part of the design.\n    •	The monolith would update immediately.
	•	The microservice might lag by milliseconds or seconds (depending on event ingestion and processing). If we compared data too early, we would catch mismatches that weren’t real. \nHow We Mitigated It:\n
	•	Introduce buffer time before comparisons (e.g., 30–60 seconds).
	•	Retry mismatches before considering them final.
	•	Split fresh vs. aged metrics so stakeholders saw the full context. \nAccepting and designing around eventual consistency prevented endless debugging loops and kept trust in the migration process intact.\n
⁠⁠Challenge #2: Validating Data Across Two Systems \nValidation wasn’t a one-off — it was a continuous, ⁠multi-layered process. \nWe used two main approaches:\n   \n Shadow Reads\n
⁠	    •	Added parallel queries to both the monolith and microservice during critical transactions.
	    •	Compared key data fields silently (without affecting user-facing behavior).
	    •	Logged any differences for investigation. This gave us live, production-grade feedback on data divergence.
⁠
  \n  ⁠Batch Reconciliation\n
⁠	    •	Ran periodic batch jobs comparing large datasets.
	    •	Focused on high-value entities and critical tables.
	    •	Highlighted drift that wasn’t visible in real-time traffic. \nShadow reads caught immediate issues; batch scans caught structural ones.\n
⁠⁠Challenge #3: Handling False Positives \nNot every mismatch was a real bug. Common false positives included:\n    •	Timestamp format differences (UTC vs local time).
	•	Optional fields behaving differently.
	•	Legacy fields populated differently. Without careful handling, these harmless differences would swamp our metrics and slow real progress. 
⁠    
\n⁠What Worked:\n
	•	Accuracy workshops: Engineers, QA, and data stakeholders sat together, reviewed mismatch samples, classified real bugs vs benign differences.
	•	Refined comparison logic: Adjusted matching rules and tolerated known differences. 
⁠
⁠These workshops helped reduce noise and allowed us to focus on true blockers to 99.9% accuracy.\n
Lessons we took away\n    •	You can’t eliminate latency in event-driven systems. You design around it.
	•	Blind trust kills migrations. Metrics, visibility, and honesty build trust.
	•	Validation needs to happen at multiple layers. Shadow reads + batch reconciliations together made our system robust.
	•	Transparency speeds up trust-building. We shared dashboards and accuracy numbers — even when they weren’t great — with leadership and engineers alike.\n
⁠Conclusion\nTrust Was Built, Not Assumed At the beginning, our accuracy was zero. There were no shortcuts — only careful validation, transparency, and iteration. 
⁠By the time we moved forward, the new system had earned trust — with measurable proof behind it, not just good intentions. 

⁠In any critical migration, that’s the real success: Trust isn’t migrated automatically. It’s rebuilt — record by record, validation by validation, and through relentless honesty in every metric.

Moniepoint Engineering

Designing trustworthy learning systems in precision medicine and the scalability gap

All Stories

Research

Unlocking Customer Insight: Applying Transformer Models to Unstructured Financial Text

Research

Federated Graph Neural Networks: Solving Real-Time Anomaly Detection Without Sacrificing Privacy

Research

Designing trustworthy learning systems in precision medicine and the scalability gap

Security & Reliability

Building Trust During System Migration: How We Reached for 99.9% Data Accuracy

Designing trustworthy learning systems in precision medicine and the scalability gap

Research

Unlocking Customer Insight: Applying Transformer Models to Unstructured Financial Text

Research

Federated Graph Neural Networks: Solving Real-Time Anomaly Detection Without Sacrificing Privacy

Research

Designing trustworthy learning systems in precision medicine and the scalability gap

Security & Reliability

Building Trust During System Migration: How We Reached for 99.9% Data Accuracy

Get more stories like this