Bobble AI Reduced Data Processing Cost by 33% by Migrating Workloads to Google Cloud

Business Opportunity

Data Migration to Reduce Cost

Bobble AI is a cutting-edge conversational media platform that enhances communication through personalized and expressive content, such as stickers, emojis, and GIFs. Leveraging AI and deep learning, Bobble AI integrates seamlessly into keyboards, messaging apps, and social platforms, enabling users to create engaging, context-aware conversations. Their focus on personalization and innovation makes Bobble AI a leader in the digital content space, transforming how people interact through mobile devices. 

Dealing with a high volume of keyboard events from its extensive user base, the client sought to optimize the performance and cost-efficiency of its data processing. Their goal was to reduce expenses while enhancing the speed and efficiency of the processing layer.

We helped them migrate their data processing from AWS EMR and Step Functions to Google Cloud Dataproc and Cloud Composer. This enabled them to leverage the high availability of spot instances on Google Cloud and reduce the cost of spark clusters with up to 90% discounts. We also helped them build a robust CI/CD pipeline which delivered a 5% Improvement in Pipeline Runtime.

The Challenge

• Data processing workloads were running on AWS using spot instances, which were frequently unavailable in required regions, leading to processing delays.
• The client preferred open-source tools over cloud provider solutions but hesitated to use managed services due to concerns about losing control over the process.
• Without a CI/CD pipeline, deployments were done manually, making the process slow and cumbersome. They were looking to implement automated deployment pipelines to streamline and accelerate the deployment process.

Our Solution

Niveus recommended migrating Bobble AI’s workloads from AWS EMR and Step Functions to Google Cloud’s Dataproc and Cloud Composer. Cloud Composer, which is based on Apache Airflow, replaced the existing Step Functions for ETL orchestration, addressing Bobble AI’s initial concerns about adopting managed services and highlighting the advantages of this transition. The aim was to leverage the high availability of spot instances on GCP, which offered up to 90% cost savings for Spark clusters. 

The migration was carefully executed by transitioning each Spark job individually, selecting optimal machine types and configurations based on the job type. At each step, the cost and runtime of the jobs on GCP were compared with those on AWS, ensuring efficiency improvements. A custom utility was built in Cloud Composer to orchestrate Dataproc cluster creation and updates. The utility prioritized spot instances, automatically switching to on-demand instances when necessary, ensuring high availability and cost efficiency.

Data from the Bobble Keyboard application flows into Google Cloud Storage (GCS) buckets, where it is processed at scheduled intervals. Cloud Composer, serving as the ETL orchestration tool, dynamically creates Dataproc clusters, optimizes spot instance usage, and submits Spark jobs. All sensitive information is securely stored in Secret Manager, and a CI/CD pipeline was built using Cloud Build for seamless deployment. Post-migration, the availability issues with spot instances were resolved, leading to faster completion of data processing.

Benefits of Leveraging Niveus’ Data Migration Expertise

• Proven Migration Framework: With extensive experience in data migrations, Niveus follows a structured, tested framework that ensures seamless transitions with minimal disruption to business operations.
• Cost Optimization Strategies: We prioritize cost-efficiency by leveraging cloud-native tools, spot instances, and tailored configurations, helping clients reduce costs while maintaining high performance.
• Performance Tuning: Our team conducts in-depth performance analyses to optimize cloud resources, ensuring faster processing times and improved system reliability post-migration.
• Automated CI/CD Pipelines: We set up automated CI/CD pipelines to streamline the deployment process, reducing manual effort, accelerating development cycles, and improving operational agility.
• Tailored Solutions: Niveus provides customized cloud solutions that align with each client’s unique needs, ensuring that infrastructure and tools are optimized for their specific workloads.
• Open-Source Expertise: Our deep understanding of open-source tools and cloud-native services enables us to offer solutions that give clients control over their stack while benefiting from cloud scalability.
• High Availability and Reliability: We ensure high availability by designing resilient architectures, utilizing features like automatic failover, and mitigating risks associated with downtime.
• Security and Compliance: Our team integrates robust security measures, including credential management and compliance solutions, to protect sensitive data and maintain regulatory standards throughout the migration process.
• Scalability and Flexibility: We design cloud architectures that are scalable and adaptable to evolving business needs, enabling clients to handle growth without compromising on performance or cost.
• Expert Guidance and Support: Niveus provides end-to-end support throughout the migration journey, from planning and execution to ongoing optimization, ensuring a smooth and successful transition to the cloud.

Results

• 33% Reduction in Data Processing Cost: The high availability of spot instances on GCP led to a 33% reduction in the cost of running Spark clusters, taking advantage of discounts of up to 90%.
• Faster and More Efficient Data Analytics: With BiqQuery Serverless API, the data can now be analyzed faster and more efficiently. 
• Modernised Data Processing: By using Data Proc and Cloud Composer we enabled a more modernized approach to data processing, ensuring future-proof strategies are in place. 
• Improved Availability and Efficiency: GCP’s consistent availability of spot instances eliminated processing delays, allowing workloads to be completed faster and more efficiently.
• CI/CD Pipeline Implementation: A new CI/CD pipeline helped speed up the development process, and reduce manual errors.
• Streamlined ETL Orchestration: Cloud Composer, based on Apache Airflow, was implemented for efficient ETL orchestration, automating the data flow and Spark job submission on Dataproc clusters.
• Enhanced Security and Compliance: Sensitive credentials were securely managed through Google Cloud Secret Manager, ensuring data security and regulatory compliance throughout the process.
• Flexibility with Open-source Tools: The adoption of open-source technologies like Apache Airflow and Spark allowed Bobble AI to maintain control over their stack while benefiting from the scalability and cost-efficiency of managed GCP services.

Power of Partnership

Niveus Solutions provided Bobble AI with the expertise needed for a seamless data processing migration from AWS to Google Cloud. By leveraging cutting-edge technologies and best practices, Niveus helped Bobble AI reduce data processing costs through the optimized use of GCP spot instances. Additionally, Niveus implemented automated CI/CD pipelines, streamlining deployments and improving efficiency. This partnership addressed Bobble AI’s concerns about managed services while enhancing their cloud capabilities for future growth.