#219 Building a Data Platform that Drives Value | Shuang Li, Group Product Manager at Box

Summary of #219 Building a Data Platform that Drives Value

Shuang Li, Group Product Manager at Box

---

Overview

Shuang Li, Group Product Manager at Box, shares insights from the multi-year journey of building Box’s data platform. The discussion highlights the challenges, key components, and strategic decisions involved in creating a scalable, reliable, and efficient data infrastructure that supports millions of users and billions of objects.

---

Key Technological Concepts & Product Features

1. What is a Data Platform?

• A high-scale data infrastructure designed to solve business problems by consolidating data from multiple sources, enabling analytics, insights, and product features.
• Before the platform, Box’s data was siloed across teams, leading to challenges in scalability, reliability, quality, and cost.

2. Core Components of a Data Platform

• Data Ingestion Pipeline: Captures data from diverse sources such as on-platform transactional data, metadata, user events, and third-party tools like SnapLogic. It must handle different formats, SLAs, and scales.
• Data Processing: Includes ETL pipelines and both batch and stream compute capabilities. Streaming is essential for near real-time use cases like anomaly detection in user activity for security.
• Data Storage & Management: Selecting appropriate tools and technologies to store and manage data efficiently, balancing performance, cost, and feature requirements.

3. Building the Platform During Cloud Migration

• The data platform was developed alongside Box’s company-wide migration to the cloud, adding complexity but enabling the use of cloud-native solutions.
• Collaboration with data engineering teams (within the go-to-market organization) and other stakeholders such as product, marketing, compliance, and customer success was critical for alignment and success.

4. Team Structure & Collaboration

• Product managers lead the data platform efforts, working closely with data platform engineers and a separate data engineering team.
• The data engineering team supports ETL pipelines and broader data needs, especially for marketing and customer success use cases.

5. Iterative Development & Milestones

• The platform development was likened to climbing a mountain with multiple peaks (milestones).
• The approach emphasized simplicity, iterative delivery, and prioritization between “lift and shift” (migrating existing workloads) and “uplift” (building optimized cloud-native architecture).

6. Technology Choices & Trade-offs

• Decisions between building in-house versus buying vendor solutions depend on feature sets, cost (licensing, engineering, maintenance), long-term scalability, vendor ecosystem, innovation speed, partnership quality, and strict security requirements.
• Early engagement with the security office is critical due to stringent compliance needs.

7. Data Quality & Observability

• Initially under-invested during migration, data quality and observability have become major focuses.
• The team restructured to separate foundational platform work from developer experience to better prioritize observability features such as data freshness, lineage, classification, and automated anomaly detection.
• Tools like Data Catalog are used for metadata management, data discovery, and lineage tracking.

8. Developer Experience & Data Democratization

• A key metric is reducing “time to value” — how quickly internal teams can onboard, explore, experiment, and move to production using the platform.
• Efforts include improving data discovery (tagging, descriptions, metadata), providing playground environments for experimentation, documentation, tutorials, and shadow environments for production-like testing.
• Data democratization focuses on making data accessible to both technical and non-technical users (e.g., product analysts), reducing time spent searching for data from weeks to much shorter periods.

9. Challenges Encountered

• Learning curve and best practices around cloud migration.
• Managing a massive, company-wide project with many stakeholders and continuously aligning goals.
• Balancing engineering team morale between foundational “lift and shift” work and innovative “uplift” features.

10. Cost Management & Scaling

• Quarterly and monthly cost forecasting considering organic growth and new use cases.
• Strategic decisions on vendor usage to optimize costs (e.g., paying a vendor to reduce ingestion volume to another vendor).
• Emphasis on the “rule of 40” metric for SaaS companies balancing revenue growth and profit margin, with data platform costs contributing to profit margin.

11. Metrics & Alignment (The “L-ther Up” Framework)

• Three levels of metrics:
  • Company-level (profitable growth)
  • Product & engineering-level (e.g., enabling new use cases like streaming)
  • Team-level (e.g., delivering streaming capability)
• This framework helps communicate impact and motivate engineers by linking their work to business outcomes.

12. Future Trends & Next Steps

• Continued investment in developer experience and tooling/frameworks to simplify data aggregation and business logic implementation for non-expert teams.
• Uplifting the log pipeline with tiered architecture supporting real-time, analytics, and compliance needs.
• Leveraging AI for data observability (automated anomaly detection, data loss, late arrival) and enhancing data discovery with natural language querying capabilities.

---

Guides, Tutorials, or Best Practices Highlighted

• Start with clear alignment across all stakeholders early and maintain it throughout the journey.
• Break down the massive project into smaller, iterative milestones with achievable goals.
• Prioritize simplicity in architecture and incremental adoption of cloud-native tools.
• Separate foundational platform work from developer experience to ensure focused investments in data quality and usability.
• Use a structured metric framework to link engineering work to business impact and maintain team morale.
• Invest in metadata management and data catalog tools to improve data discovery and democratization.
• Engage security teams early when selecting vendors to avoid delays.

---

Main Speaker / Source

• Shuang Li – Group Product Manager at Box, leading the data platform team.

---

End of Summary