Unlocking Scalable Data Platforms in Southeast Asia through Data Mesh and Google Cloud

In 2022, over 460 million individuals in Southeast Asia accessed the Internet, representing an 80 percent penetration rate, as reported by ASEAN.org. As Southeast Asia undergoes a digital transformation, businesses in Singapore, Indonesia, Vietnam, and the surrounding regions are creating and leveraging more data than ever. This increase in data presents both opportunities and challenges, especially concerning scalability, data ownership, and immediate insights. In this blog, we will explore data mesh best practices in Singapore, the role of scalable data platforms in Southeast Asia, the benefits of data mesh for SEA enterprises and Google Cloud’s data mesh best practices.

The digital economy in ASEAN is growing swiftly and is expected to reach USD 600 billion by 2030, leading to changes in the requirements for data infrastructure. The rising need for local, scalable, and resilient data centers in the region is driven by stringent data sovereignty laws that require data to be stored and processed locally. Rising geopolitical tensions around China and Hong Kong, along with a waning attractiveness of the area for data center operations, make ASEAN a more secure and appealing choice for worldwide digital infrastructure. Due to limitations in land and energy resources in Singapore, investments are shifting towards more cost-effective and high-growth prospects in Malaysia, Indonesia, and Thailand, which provide strong infrastructure and enticing development incentives. As this shift occurs, a modern methodology for data architecture is surfacing that decentralizes data ownership and turns data into a product, enabling business domains to manage their data pipelines.

The Role of Scalable Data Platforms in Southeast Asia  

It is anticipated that the SEA’s regional companies will allocate USD 4 billion towards cloud computing by 2026, with a compound annual growth rate of 33%. Currently, 40% of these companies have implemented cloud technology initiatives, while governments are actively promoting digital transformation. Consequently, there is a growing demand for local and secure data centers and scalable data platforms. 

What is Data Mesh

Legacy, centralized data architectures often hit a wall when organizations grow too fast. Centralized teams become bottlenecks, data gets siloed, and time-to-insight plummets. As Southeast Asian enterprises expand geographically and across business units, these issues become amplified.

To solve these headaches, Zhamak Dehghani introduced Data Mesh as a paradigm. In contrast to conventional architectures that centralize everything, Data Mesh decentralizes data ownership and responsibility throughout the organization—something similar to how microservices revolutionized software delivery.

Figure 1: The Four Principles of Data Mesh Implementation

It is founded on four important principles:

• Principle 1: Domain-Driven Data Ownership
  • Associates accountability with the business rather than technology. 
  • Calls for a transformation in data methodology and a fresh data architecture: a self-serve data platform functioning on a federated governance framework. 
  • Advantages include defined ownership and boundaries, scalability, improved data quality, and quicker time to market.
• Principle 2: Data as a Product
  • Teams need constant data sharing, 
  • Each team defines their own and specifies the data they produce and consume from others. This brings in data products as an idea. 
  • A data product is a self-contained unit of data that solves a business problem.
  • Data products can be simple (tables or reports) or complex (machine learning models).
  • They achieve this through interfaces, contracts, versions, and access rules.
  • The product should mask sensitive data fields from personnel without clear access.
• Principle 3: Self-Serve Data Platform
  • Domain data teams take charge of their own data products concerning ingestion, transformation, data quality verification, and analytics. 
  • A self-serve data platform comprises tools for data ingestion, transformation, storage, cleaning, testing, and analysis. 
• Principle 4: Federated Computational Governance
  • Within a data mesh structure, while domain teams are responsible for their data products, the data platform and corporate data governance team oversee and manage compliance centrally. 
  • Federated computational governance facilitates scalable data governance, automating policy implementation and minimizing the manual effort needed to comply with data regulations.

Benefits of Data Mesh for SEA Enterprises

Embracing Data Mesh allows enterprises in Southeast Asia to:

• Accelerate time-to-insight by reducing centralized data bottlenecks.
• Enhance business agility by empowering domain teams.
• Implement a “data-as-a-product” mindset for improved data quality.
• Facilitate cross-border scalability through federated governance and localized control.
• Address regional regulatory requirements with flexibility.

As numerous organizations in SEA modernize their practices, adopting a Data Mesh framework enables them to expand their data infrastructure in a manner that supports both growth and compliance.

Google Cloud Data Mesh Best Practices in Singapore and SEA

• Organize Teams by Business Domains
  • Structure teams according to business areas (e.g., sales, marketing, finance).
  • Delineate and manage domains with Google Cloud folders and projects.
  • Apply IAM roles and policies at the folder or project level to establish ownership of domains.
• View Data as a Product
  • Designate product owners to oversee datasets, emphasizing quality, documentation, and ease of discovery.
  • Store data in BigQuery, Cloud Storage, or Dataplex-managed zones tailored to specific use cases.
  • Define SLAs/SLOs for data products (e.g., freshness, availability).
  • Utilize Data Catalog for managing metadata and enhancing the discoverability of data products.
• Enable Self-Service Data Infrastructure
  • Develop a platform layer comprising components such as:
  • BigQuery for analytics and querying,
  • Dataflow or Dataproc for data processing,
  • Cloud Composer or Workflows for orchestration,
  • Dataplex for cohesive data management.
  • Provide reusable templates, pipelines, and configurations for domain teams’ use.
  • Employ Vertex AI and Looker for AI/ML and BI tools when needed.
• Implement Federated Governance
  • Establish central data governance protocols (e.g., access control, lineage, quality).
  • Utilize Dataplex and Data Catalog for centralized enforcement of metadata and policies.
  • Access and utilization tracking  can be done via Cloud Audit Logs and BigQuery usage statistics
  • Enforce Data Loss Prevention (DLP) measures for the scanning and classification of sensitive data.
• Security and Access Recommendations
  • Use IAM Conditions to manage authorization based on context (time, location).
  • Implement VPC Service Controls for enhanced perimeter security.
  • Utilize Cloud IAM and resource hierarchy to maintain least privilege access.
• Monitoring and Observability Setup
  • Implement Cloud Monitoring and Cloud Logging for insights into infrastructure and pipeline status.
  • Use Dataplex for validating data quality and tracking data lineage.
  • Create dashboards to keep track of metrics at both the domain and platform levels.

• Automation and Continuous Integration/Continuous Deployment
  • Terraform or Google Cloud Deployment Manager for automating infrastructure setup 
  • Employ Cloud Build or GitHub Actions for CI/CD processes for data pipelines and infrastructure.

Conclusion 

Southeast Asia is experiencing a significant digital expansion, with businesses producing and utilizing more data than ever before. This trend presents both opportunities and challenges, particularly regarding scalability, data ownership, and real-time data access. Ongoing geopolitical tensions between Hong Kong and China, along with the declining appeal of Singapore as a location for data centers, enhance ASEAN’s position as a viable source for global digital infrastructure. To tackle these challenges, a contemporary data architecture is essential, one that separates data ownership and empowers specific business domains. Data Mesh, which focuses on domain-specific data ownership, productized data, a self-service data platform, and federated governance of computational resources, offers benefits such as faster insights, increased business flexibility, improved data quality, cross-border scalability, and better management of regulatory challenges.