Case Study
The Client
The client is an Indian-origin multinational automotive manufacturing corporation, headquartered in Mumbai. The company is the leading vehicle manufacturer in India as well as the largest manufacturer of tractors in the world by volume. Their tractor models are known for being powerful, having a high lift-capacity, dual-acting power steering, and more.
Project Objective
Business Value – With our BigQuery optimization solution, Niveus helped the client to improve the quality of their application, integrating cost efficiency, and scaling their VMs for better performance.
The client has a frontend application that showcases the summarized data that are gathered from all the tractors for farm operators. They were looking to solve their issue of high infrastructure costs for this application, due to multiple VMs being run or due to multiple read/writes involved in the algorithm’s steps. Additionally, the cost of maintaining and updating multiple VMs can be a challenge for organizations. In order to offset these costs, the client chose to use cloud-based solutions that allow them to pay only for the resources they use.
Business Solution
Niveus created a data ingestion system that acquires data from a variety of sources and then stores it in a centralized location. This is programmed to receive data packets every 30 seconds from each tractor while it is running, and every 2 hours when it is not. A calculation algorithm runs every 15 minutes, processing this raw data. Niveus worked on the BigQuery optimization algorithm and used the Carto API to bring down the cost to 90% of the original consumption. Scalability is dramatically improved with all infrastructure management by automatically scaling up and down from zero, almost instantaneously using Cloud Run.
Implementation
- Raw data is stored in BigQuery
- The Dataflow job fetches all the data that was processed at once using BigQuery optimization and stores it in the pipeline for the calculation
- The intermediate read/writes happen in the pipeline of the Dataflow
- The summarized data is stored either back in BigQuery or directly to the BigTable
- The data in BigTable is being used by the frontend application for farm operators
The Impact
Cost reduced to 90% of the original consumption
VM scalability is dramatically improved with automation
Technology Stack
IoT core
Cloud Pub/Sub
BigQuery
Cloud Dataflow
Cloud Run
Kubernetes Engine
Cloud Memorystore
