In a data-driven world, PayPal’s ability to deliver timely and actionable insights is central to staying ahead. At PayPal, data powers everything from fraud detection to user experience enhancements. Data is also central to unleashing the potential of agentic solutions and experiences. Over time, though, our analytics environment had become a complex ecosystem of various technologies and solutions assembled on-premise to address growing demands. While this approach supported our needs at the time, it began presenting new challenges to scale and maintain.
Due to expedited growth and acquisitions, our data analytics platform gradually turned into an uneven landscape. Each new platform or integration addressed a specific business need, but together, they increased operational overhead and introduced performance blockages. Scalability became increasingly difficult, and time-to-insight slowed as processes grew more complex. PayPal’s legacy data analytics platform was powerful—handling petabytes daily—but it was also increasingly rigid following rapid growth.
Scaling up during peak retail events or global launches meant months of planning, slow manual provisioning of hardware, and too often, a compromise between speed and cost. As PayPal continued to scale globally, we recognized the need for a streamlined, unified infrastructure to drive data efficiency and accelerate innovation. To overcome these obstacles, we migrated our analytics workloads from legacy Hadoop on-premise platforms to Google’s Managed Service for Apache Spark .
Rapid provisioning and elastic scaling: Managed Spark enabled us to deploy clusters in minutes and scale based on processing needs, eliminating lengthy setup and idle resource costs. Unified infrastructure: Standardizing on Apache Spark created consistency across teams while leveraging Managed Service for Apache Spark and other managed services reduced operational complexity. Seamless integration: Native hooks into Google Cloud Storage (GCS), BigQuery , and other Google Cloud services streamlined end-to-end data movement.