Blog Posts by Andy Feng

  • Storm and Hadoop: Convergence of Big-Data and Low-Latency Processing

    At Yahoo!, Hadoop plays a central role in providing personalized experiences for our users and creating value for our advertisers. To serve Yahoo!’s emerging business needs, the Cloud Engineering Group is working on a next generation platform that enables the convergence of big-data and low-latency processing.

    Figure 1. Personalization based on User Interests

    Figure 1. Personalization based on User Interests

    Yahoo! is enhancing its web properties and mobile applications to provide its users personalized experience based on interest profiles. To compute user interest, we process billions of events from our over 700 million users, and analyze 2.2 billion content every day. Since users' change interest over time, we need to update user profiles to reflect their current interests. Figure 1 illustrates a conceptual architecture that describes how low-latency processing and batch processing are leveraged to update users' interest profile for personalization.

    Figure 2. Convergence of batch and low-latency processing

    Figure 2. Convergence of batch and low-latency processing

    Enabling low-latency

    Read More »from Storm and Hadoop: Convergence of Big-Data and Low-Latency Processing