Accelerating Report Generation with Optimized Data Queries

Business Opportunity

Faster Dashboard Reports Generation

The client, a leading telecommunications provider in Malaysia, operates in a competitive market driven by usage-based billing and operator portability. With a 2023 service revenue of USD 1.83 billion, timely decision-making is crucial for sustaining this scale. However, batch processing of data caused delays, with reports reaching senior management only by the third day. This latency hindered their ability to respond promptly in a market demanding agility.

Niveus optimized the client’s data lake and warehouse by converting DAGs into efficient BigQuery queries and optimizing query structures. This optimization resulted in an 88% reduction in TAT for report generation, thereby accelerating data-backed decision-making.

The Challenge

The client encountered three key challenges in optimizing their data operations:

• Latency in Data Pipelines: Previously, the process involved data generated on a given day being batch processed and the reports generated were made available to senior management only by the third day, causing delays in decision-making. 
• Limited Tools for Data Optimization: A centralized approach for validating and refining queries has not yet been implemented, leading to extended runtimes during data ingestion and transformation processes.
• Scalability Constraints: The existing workflows operated without full utilization of parallel processing, limiting scalability and occasionally leading to increased latency and operational complexity.

Our Solution

Niveus implemented a comprehensive optimization strategy to enhance the client’s data lake and warehouse operations. This included simplifying compute-heavy workflows in Directed Acyclic Graphs (DAGs), which are structures used to represent and execute sequences of tasks without circular dependencies, by converting them into efficient BigQuery queries. Additionally, query structures were optimized using techniques like selective column retrieval, partitioning, clustering, and filter reordering.

Scalability improvements involved implementing a slot consumption dashboard for infrastructure sizing insights and transitioning select workloads to sensor-based triggers, enabling efficient scheduling and resource utilization. This holistic solution significantly improved data pipeline performance, scalability, and decision-making agility.

Results

The optimization strategy delivered significant improvements in performance and efficiency, including:

• 88% Reduction in Report Generation for 5 Key Dashboards: With completion now occurring within 8 hours of source data availability instead of 3 days, ensuring timely insights for business decision-making.
• Optimized DAG Runtime to Under 1 Hour: Reduced the runtime of all Dataflow DAGs to under 1 hour wherever feasible, enabling faster data processing and streamlined workflows. 
• Enhanced Data Pipeline Performance: Streamlined workflows and optimized BigQuery queries significantly reduced processing times, ensuring faster data availability and improved operational efficiency.
• Improved Scalability: Implemented advanced infrastructure monitoring and sensor-based triggers, enabling the system to scale effortlessly with growing data demands.
• Increased Decision-Making Agility: Delivered near real-time insights through Looker Studio dashboards, empowering senior management with timely, actionable data for better decision-making.

These results highlight the effectiveness of the optimization strategy in accelerating data delivery and enhancing operational performance.

Power of Partnership

Niveus brought extensive expertise in cloud architecture and data optimization to this implementation, leveraging a deep understanding of BigQuery, Looker Studio, and scalable data management solutions. Our team designed and executed the transformation of compute-heavy workflows into optimized BigQuery queries, significantly enhancing performance. By implementing advanced query optimization techniques and near real-time monitoring tools, Niveus ensured seamless data pipeline execution. Additionally, our proactive approach in scaling infrastructure and implementing sensor-based triggers for scheduling further streamlined operations, delivering a robust and agile solution that aligned with the client’s evolving needs.

Benefits of Niveus’ Data Optimization Solutions

Niveus’ robust data lake and warehouse optimization strategies can transform data management into a strategic enabler, driving innovation, agility, and operational excellence.

• Enhanced Performance and Scalability: Our optimization strategy ensures that data lakes and warehouses can handle large-scale data processing and query execution efficiently, even as data volumes grow. This enables businesses to scale seamlessly while maintaining consistent performance across analytics and reporting workloads.
• Cost Efficiency: Through our effective resource management and cost control measures- such as tiered storage, right-sizing compute resources, and automated data lifecycle policies – organizations can reduce operational expenses while maximizing value from their data infrastructure.
• Improved Data Quality and Accessibility: By optimizing data pipelines, businesses can ensure accurate, consistent, and clean data. Additionally, our well-structured storage and efficient query mechanisms make it easier for teams to access the data they need, fostering faster and more informed decision-making.
• Faster Time to Insights: By reducing latency in data processing and query execution, our optimization strategies empower businesses to gain actionable insights in near real-time. This agility is critical in industries where timely decisions drive competitive advantage.
• Robust Security and Compliance: Our security practices include implementing advanced security controls, encryption, and access policies. This ensures that sensitive data is protected and that the organization adheres to regulatory standards