Transforming a Tier 1 Australian Financial Organisation’s Modern Data Platform with a 300% Performance Uplift
Cevo successfully modernised a prominent Australian financial services group’s big data analysis platform, transitioning from Cloudera to Spark on Amazon EKS. The phased approach included tailored clusters for distinct business units. The new modern data platform has resulted in a 300% performance boost for critical data jobs, automatic scalability, streamlined operations, cost optimisation and enhanced governance, enabling agile decision-making and efficient management across business units.
300% performance uplift
For critical data jobs
Insights
Cevo NexusTM solution
Financial Services
Industry
The customer is a prominent Australian financial services group, offering banking, superannuation, wealth management and advisory services globally.
Business challenge
The financial organisation faced a critical need to modernise its big data analysis platform due to scalability and cost optimisation challenges with the existing Cloudera-based system. Large jobs were experiencing prolonged execution times and, in some cases, exceeded existing infrastructure capacity, causing unnecessary costs. The one-size-fits-all approach from the corporate operation group also limited individual business units in tailoring the platform to their specific needs.
Distinct data clusters were essential for various business units. The financial teams required a high-performance environment for upper regulatory reports, while the risk management teams required a platform that adhered to strict regulations and compliance standards for secure and controlled access to critical data.
The financial organisation sought improved scalability and efficiency while creating tailored clusters for each business unit. They wanted to reduce costs associated with the Cloudera platform while maintaining overall governance and central management.
Solution
Cevo partnered with the financial organisation to modernise its big data analysis platform by transitioning to Spark on Amazon EKS. The project adopted a phased approach to address the distinct needs of each business unit.
Working closely with the financial organisation, Cevo gained insights into the existing corporate data hub environment, its functions, and design requirements for the target state architecture. Cevo architected both general purpose and dedicated clusters. The general purpose cluster serves smaller teams for ad-hoc analytics, reducing costs, while dedicated clusters meet the unique performance and compliance requirements of the financial and risk management teams.
The solution enabled controlled data access, and seamless integration between data lakes, and featured EKS clusters designed for autoscaling, cost optimisation, and governance through role-based access control. Security measures included Aquascans for container vulnerability scanning.
The second phase focused on complex integrations, implementing Grafana and Prometheus for real-time metrics, a unified dashboard for real-time insights, and Spline for data lineage visualisation. A GitOps approach with Helm charts empowered business units, and integration with Apache Airflow facilitated the migration from the legacy Cloudera platform to Spark clusters on EKS. A scheduler was developed for automation, including notifications, reports, and dry run options. The platform is integrated with the Hive metastore to manage metadata.
Building the Jupiter hub as a pod on the cluster with AD login gave the data science team secure access. The design also accommodated a personalised approach for integrating Conda packages, enabling the exploration of specialised packages for data science and ML projects.
Automation, leveraging AWS Lambda, AWS CloudFormation, AWS Systems Manager, and AWS Event Bridge, ensures efficient rightsizing and resource management. Cevo’s engineers collaborated closely with the financial organisation, adopting an agile methodology for regular communication, quick issue resolution, and efficient implementation of optimisation strategies aligned with evolving needs.
Outcomes
The financial organisation successfully achieved its target state of a modern data platform, resulting in several significant benefits:
- Improved performance: The new data platform helped the financial organisation realise a 300% performance improvement for significant data jobs, due to autoscaling EKS clusters and Spark version upgrades.
- Enhanced scalability and adaptability: The new platform enabled efficient resource usage, automatically adapting to changing capacity needs for varying job sizes.
- Modernised operations: Streamlined operations by reducing noise from multiple business units sharing infrastructure.
- Unified metrics dashboard: The unified metrics dashboard provides a single pane of glass for usage and capacity, facilitating informed decision-making across business units.
- Improved data understanding: The platform enhanced visibility into data movement between different systems, allowing the company better insights into each business unit.
- Reduced licensing costs: The new modern data platform easily allows for cost-effective resource scaling. Transitioning away from the previous licensed vendor also resulted in reduced licensing costs.
- Enhanced governance and security: Implemented governance and monitoring tools for enhanced security and compliance.
- Improved platform management: The shared, flexible and tailored infrastructure significantly benefited both platform teams and individual business units, making platform management seamless.
Enjoyed this customer story?
Share it with your network!
You may also like
