In investment management, speed, and accuracy matter. Firms collect vast amounts of financial data but often struggle to turn that data into valuable insights. The problem isn't a lack of data; it's how the firm manages it. Analysts can spend up to 80% of their time cleaning and moving data instead of analyzing it.
One reinsurance and investment firm faced this exact problem. Working with DataArt, the firm redesigned its data systems using a federated data mesh approach. A system lets teams own and manage their data while keeping everything connected and consistent. This approach helped the firm make better decisions faster – and at a lower cost.
What Is Data Mesh?
Traditional data systems centralize all data, slowing things down and creating bottlenecks. Data mesh changes give individual teams control over their data while maintaining overall standards.
Think of it as breaking data management into smaller pieces. Each team is responsible for its own "data product," but all teams follow shared rules.
Together with DataArt, our client built their solution on four key ideas:
- Domain Ownership: Teams manage the data they know best
- Data as a Product: Treat datasets like products designed for others to use easily
- Self-Service Tools: Teams access data without waiting on IT
- Shared Governance: Keep standards and controls without slowing innovation
This balance is critical for a firm with complex data and strict regulations.
Fixing Data Problems Step-by-Step
Before the change, the firm's data was stuck in silos, with inconsistent datasets slowing research. Analysts were bogged down fixing data flows instead of analyzing markets.
DataArt focused on four areas to fix this:
1. Updating the Data Architecture
Our team designed and implemented a new data architecture based on a lakehouse model, which balances data lakes' flexibility and data warehouses' structured reliability. At its core, the system uses AWS S3 for storage. At the same time, Dremio serves as the query engine, enabling the firm to process data faster and cost-effectively than traditional cloud data warehouses like Snowflake.
We integrated Apache Iceberg to support critical use cases such as backtesting and audits. This technology allows the firm to access historical versions of its data, providing a "time travel" feature that shows precisely how data looked at any moment.
The data pipelines were built using ELT (Extract, Load, Transform) principles, with Apache Airflow managing scheduled batch tasks and Kafka Connect handling real-time data streams. This combination provides the firm with a robust, scalable system that delivers timely and reliable data across the organization.
(In simple terms: "Time travel" means the firm can see what data looked like at any point in the past.)
2. Focusing on Data Quality
To improve data quality, our team integrated Soda, a tool that profiles data and identifies anomalies early in the process. Alongside this, the firm is piloting Great Expectations to validate data as it arrives – not in real-time down to the millisecond, but close enough to catch issues promptly.
While there was some internal debate about overlapping functionality between these tools, the team is taking a measured approach to determine the right balance. Meanwhile, Datadog dashboards offer continuous, real-time visibility into data quality across all pipelines, helping maintain trust in the system.
3. Supporting Quantitative Analysts
The analysts' focus is on market analysis and model building – not troubleshooting data pipelines. Data flows directly into familiar tools like Jupyter Notebooks, Power BI, and Tableau to support this.
We standardized data models to align with the investment teams' operations, ensuring consistency and ease of use. Metadata catalogs powered by DataHub and AWS Glue keep data well organized, with plans to integrate Apache Polaris for even better management. Automated testing runs in the background to maintain data reliability, allowing analysts to dive straight into their work without delay.
4. Making Data Accessible Through Natural Language
To make data accessible to non-technical users, the firm introduced Gen BI, a tool that lets people query data using plain English. After exploring several options, the firm selected a solution that runs securely offline, ensuring sensitive data stays protected.
Governance rules are built into the system so users can freely explore data without risking compliance breaches.
What Changed
This overhaul delivered precise results:
- Data quality improved, building trust across teams.
- Infrastructure costs dropped by 60 to 80 percent.
- Analysts shifted focus from data plumbing to strategy.
- More people can access data, leading to faster new ideas.
Instead of chasing the fastest execution speeds, the firm first focused on building a solid foundation and adding AI and analytics.
What Others Can Learn
If your firm struggles with scattered data and the pressure to add AI responsibly, this case offers clear guidance:
- Design a federated system that distributes data ownership.
- Validate data as early as possible in the process.
- Give domain experts the tools to work independently.
- Add AI features step-by-step, keeping controls in place.
- Keep data discoverable, consistent, and monitored.
Success comes down to being ready to use your data – not just having more of it. This work with DataArt shows that thoughtful architecture, practical tools, and disciplined data management move the needle.
