Redefining Data Strategy: Optimized Analytics with BigQuery

Business Opportunity

Data Modernization with BigQuery

The client is India’s largest car-sharing marketplace, operating in India, Indonesia, and Egypt. It connects vehicle owners with renters through a tech-driven platform, offering over 20,000 cars. Employing 250+ people, the client operates in 45+ cities. 

They aimed to migrate from AWS Redshift to GCP BigQuery to enhance performance, optimize resources, and improve data governance for more efficient and cost-effective analytics.

The Niveus team modernized their AWS Redshift and Athena workloads by migrating to GCP BigQuery, consolidating data into a single platform for better performance and scalability. This unified source of truth ensures consistent, accurate, and accessible data across the organization.

The Challenge

• Data Volume and Resource Optimization: The client’s AWS Redshift data warehouse struggled to handle increasing data volumes and optimize cluster resources effectively.
• High Cluster Utilization and Performance Slowdowns: The Redshift cluster utilization frequently exceeded 90%, leading to performance slowdowns, particularly during spikes in usage.
• Complex Issue Tracing and Data Access Tracking: Tracing upstream issues and tracking data access was complex, prompting the need for better data lineage and user activity tracking.

Our Solution

The Niveus team successfully modernized the existing AWS Redshift and Athena workloads by migrating to GCP BigQuery using a lift-and-shift approach. This involved transferring Athena tables to S3 buckets as external tables, consolidating data into a single enterprise platform, and migrating Redshift data to BigQuery. 

Additionally, the team implemented incremental data ingestion from Freshdesk (SaaS platform) and integrated application data from MySQL, PostgreSQL, and MongoDB. They also migrated PySpark jobs from AWS EMR to GCP Dataproc, designed a three-layer architecture for the BigQuery Data Warehouse, and translated and optimized queries for BigQuery. Finally, the reporting layer was transitioned to Compute Engine and Tableau, ensuring an efficient and scalable data analytics platform. 

With all data now consolidated into a single platform, the client achieved a unified source of truth, ensuring consistent, accurate, and accessible data across the organization.

Results

• 2.3 TB of Historical Data Moved: Successfully transferred over 2.11TB of historical data to the new platform, which helped in building a single source of truth.
• 3,238 Redshift Tables Migrated: Migrated 3,238 Redshift tables to GCP BigQuery for enhanced performance and scalability.
• 1,802 Athena Tables Migrated: Transferred 1,802 Athena tables to the new architecture, ensuring seamless data access.
• 27.9 TB of Data Transferred: Migrated 23 S3 buckets with 27.9 TB of data, consolidating it into the new unified platform, boosting performance with centralized data access.
• 27 Pipeline Jobs Migrated: Successfully migrated 27 pipeline jobs to streamline data processing on the new system.
• 6 PySpark Jobs Migrated: Migrated 6 PySpark jobs from AWS EMR to GCP Dataproc, optimizing data workflows.

Power of Partnership

Niveus brought a wealth of expertise and strategic insight to the client’s data platform transformation. Our team’s strong understanding of data management, resource optimization, and process streamlining enabled us to deliver a seamless migration from AWS Redshift to GCP BigQuery. By leveraging our ability to drive collaboration and manage complex projects, we ensured the successful consolidation of the client’s data into a single, unified platform. 

Our focus on delivering reliable, accurate, and accessible data was key in helping the client achieve a single source of truth. Through our proactive approach, we demonstrated our capability to tackle challenges with precision and efficiency, building a strong, lasting partnership that contributed to the client’s enhanced data analytics capabilities.

Benefits of Migrating to BigQuery with Niveus

• Expert Assessment & Planning: Tailored end-to-end migration strategies built on deep discovery and ROI analysis with minimal disruption using proven tools and frameworks.
• Cost Optimization Strategies: Smart design and tuning to maximize performance while minimizing BigQuery costs.
• Performance Engineering: High-efficiency data pipelines and query optimization for large-scale analytics.
• Security & Governance Enablement: Enterprise-grade security setup with strong governance and compliance controls.
• Domain Expertise & Industry Templates: Pre-built accelerators and industry-specific solutions for faster time-to-value.
• Post-Migration Support & Scaling: Continued optimization and scaling support to future-proof your data platform.