DataArt Case Study: Solución de Análisis y Reportes Basada en la Nube

Business Challenge

The client’s existing solution relied on an obsolete diversified technology stack, leading to excessive operational costs and complex maintenance. The platform had a fragile data architecture and legacy hardware dependency, which caused additional issues in maintaining an already expensive data center and licenses.

The legacy platform failed to meet the high-performance demands, and data processing was slow. The POS transaction pipeline took up to two days to process one month of data.

The client was looking to transform the existing platform to minimize the overall operating costs, migrate from a cost-inefficient data center to cloud data services, and optimize the pipeline performance for historical data processing to ensure results within a few hours.

Meeting the Challenge

The overall transformation was affected by a lack of documentation, resources, and people who could answer questions. Essentially, DataArt’s team leveraged its data consulting services expertise to treat the entire solution upgrade as a reverse engineering exercise. This way, the team meticulously analyzed the source code, data architecture, implementation details and configuration, and the deployment model, all without any technical insight.

The following approach was used to resolve the issues:

Cloud Enablement and DevOps Streamlining

Data processing, storage, and all services were migrated to the AWS (Amazon Web Services) environment. Data processing was done by deploying the solution into a DC/OS cluster based on AWS resources. All deployable components are now delivered in Docker containers to ship them to a newly created GuestMetrics Ops team and affect the deployment process more easily.

Data Management Strategy

Instead of maintaining a Hive cluster-based solution using a mix of Java, Bash, SQL, HDFS, and Hive, the entire solution was converted into a portable data processing pipeline based on Apache Spark. This made it possible to eliminate dependencies and, more importantly, run the data pipeline continuously every six hours instead of once a month. As a result, the data pipeline met the client’s platform requirements.

Dashboard and User Management Solution

The legacy platform was transformed using R and a Shiny server for visualizing data. The development team decided to keep the platform as a subcomponent of a more scalable and easily extendable solution due to the complex and time-costly migration of the Dashboard system. This made it possible to enable SSO integration and customer data software customization based on individual client needs.

Business Benefits

The DataArt team transformed the pipeline into a new, fast, cost-effective, and optimized technological solution capable of quickly processing transactions. The solution was migrated to AWS, thus eliminating the hardware dependency and minimizing maintenance costs. Supplementary scalability was achieved using docker containerization.

The earlier, slow data processing system prevented the client from developing their business and from providing their customers with data promptly.

The new and more effective data processing pipeline enabled by cloud data services helped the client achieve their business goals and successfully meet customer requirements.

Conclusion

GuestMetrics’ journey highlights the importance of partnering with a data analytics consulting firm to modernize and optimize data platforms. By leveraging cloud data services and data migration solutions, we enabled a scalable, cost-effective, and efficient solution that meets today’s business demands.