The Powerful Technology behind IntoNow 3.0: Ambient Computing

Today we released IntoNow 3.0 from Yahoo!. It’s a major reengineering of our award-winning second-screen experience.

The new features in 3.0 are incredible, and set IntoNow far ahead of the pack. We were able to accomplish this new release through Yahoo!’s deep science and scale. For the last few months, I’ve been embedded in the IntoNow team, and I thought you’d enjoy a behind-the-scenes look at the powerful technology behind IntoNow 3.0:

Ambient computing

The second-screen experience is a little different from a typical PC or mobile experience because you have to account for the user multi-tasking. One of our key goals is to implement “ambient computing,” where the user does not have to actively initiate any action and the computation happens in the background. The results are steadily streamed to the user. There are several features in the IntoNow app that are driven by this paradigm:

TV and Music Sync

While you’re watching a TV program, one of components of IntoNow 3.0 is to surface content related to the show. For example, on Cramer’s “Mad Money,” entering Cramer's rapid-fire stock advice into your tablet/smartphone previously required the user to take multiple steps. However, with ambient computing, the stock will automatically surface in IntoNow. One click takes you to the Yahoo! Finance page for the stock profile, providing the user with the right information at the right time, instantaneously. That’s the power of ambient computing.

In the background, the TV Sync system is continuously processing the TV closed captions from 100s of channels, in real-time, extracting meaningful entities (people, places, stock tickers, sports teams, Olympics athletes, etc.), cross-referencing them with news articles, tweets, and other content. It is identifying interesting content at the same time.

The content you see surfaced in this manner uses the Yahoo! Content Acquisition Platform (CAP), Yahoo! Search, Hadoop, MObStor (our massively distributed object store), YQL (Yahoo Query Language) and others. We use state-of-the-art entity recognition and tagging technologies, built over the years at Yahoo. All of this happens in milliseconds, giving you the freshest content instantaneously.

Music Sync - Automatically identify the music playing during a TV showMusic Sync - Automatically identify the music playing during a TV show

“Music Sync” is the first technology to use Ambient Computing to automatically identify the music played during a show – whether a live performance, featured song or even background music. If you’re watching American Idol or Breaking Bad, and a snippet of a song comes on, by the time you fire up your favorite music-identification app and hit the tag button, the song may be over, and you’re left wondering, what was that that song? Or, it may not recognize the song if it wasn’t played by the original artist.

With our Music Sync feature, the song will automatically be surfaced to you, along with a link to its YouTube video and the iTunes store to buy the album or the song. And at the end of the episode, all of the music identified is available for you to browse.

CapIt - Capture and caption TV momentsCapIt - Capture and caption TV moments

CapIt

CapIt delivers industry-first technology that gives people the ability to capture and caption “did you see that?” TV moments like a breaking news story, a red carpet dazzler or even a funny scene.

Simply tap a button and the app will instantly synchronize and deliver still images of the program that’s shown on TV. Then you can select the image you want, add a caption, and share with friends on Facebook, Twitter or IntoNow. You can also earn loads of karma on Reddit.

The ability to do this at scale on hundreds of TV channels instantly is one example of Yahoo!’s scale on display. “CapIt” generates more than 13 million images per day, each day. When you hit the “CapIt” button, the right set of images are instantly pulled up and served to you.

All of these new features are made possible thanks to some of Yahoo!’s core technologies and scale. I work in Yahoo! Labs (and I’m embedded with the IntoNow team), where we work on deep Science. Yahoo!’s powerful infrastructure provides the Scale. The S’s come together to deliver innovative products at a blistering pace.