Today we released IntoNow 3.0 from Yahoo!. Its a major reengineering of our award-winning second-screen experience.
The new features in 3.0 are incredible, and set IntoNow far ahead of the pack. We were able to accomplish this new release through Yahoo!s deep science and scale. For the last few months, Ive been embedded in the IntoNow team, and I thought youd enjoy a behind-the-scenes look at the powerful technology behind IntoNow 3.0:
The second-screen experience is a little different from a typical PC or mobile experience because you have to account for the user multi-tasking. One of our key goals is to implement ambient computing, where the user does not have to actively initiate any action and the computation happens in the background. The results are steadily streamed to the user. There are several features in the IntoNow app that are driven by this paradigm:
TV and Music Sync
While youre watching a TV program, one of components of IntoNow 3.0 is to surface content related to the show. For example, on Cramers Mad Money, entering Cramer's rapid-fire stock advice into your tablet/smartphone previously required the user to take multiple steps. However, with ambient computing, the stock will automatically surface in IntoNow. One click takes you to the Yahoo! Finance page for the stock profile, providing the user with the right information at the right time, instantaneously. Thats the power of ambient computing.
In the background, the TV Sync system is continuously processing the TV closed captions from 100s of channels, in real-time, extracting meaningful entities (people, places, stock tickers, sports teams, Olympics athletes, etc.), cross-referencing them with news articles, tweets, and other content. It is identifying interesting content at the same time.
The content you see surfaced in this manner uses the Yahoo! Content Acquisition Platform (CAP), Yahoo! Search, Hadoop, MObStor (our massively distributed object store), YQL (Yahoo Query Language) and others. We use state-of-the-art entity recognition and tagging technologies, built over the years at Yahoo. All of this happens in milliseconds, giving you the freshest content instantaneously.
Music Sync - Automatically identify the music playing during a TV show
Music Sync is the first technology to use Ambient Computing to automatically identify the music played during a show whether a live performance, featured song or even background music. If youre watching American Idol or Breaking Bad, and a snippet of a song comes on, by the time you fire up your favorite music-identification app and hit the tag button, the song may be over, and youre left wondering, what was that that song? Or, it may not recognize the song if it wasnt played by the original artist.
With our Music Sync feature, the song will automatically be surfaced to you, along with a link to its YouTube video and the iTunes store to buy the album or the song. And at the end of the episode, all of the music identified is available for you to browse.
CapIt - Capture and caption TV moments
CapIt delivers industry-first technology that gives people the ability to capture and caption did you see that? TV moments like a breaking news story, a red carpet dazzler or even a funny scene.
Simply tap a button and the app will instantly synchronize and deliver still images of the program thats shown on TV. Then you can select the image you want, add a caption, and share with friends on Facebook, Twitter or IntoNow. You can also earn loads of karma on Reddit.
The ability to do this at scale on hundreds of TV channels instantly is one example of Yahoo!s scale on display. CapIt generates more than 13 million images per day, each day. When you hit the CapIt button, the right set of images are instantly pulled up and served to you.
All of these new features are made possible thanks to some of Yahoo!s core technologies and scale. I work in Yahoo! Labs (and Im embedded with the IntoNow team), where we work on deep Science. Yahoo!s powerful infrastructure provides the Scale. The Ss come together to deliver innovative products at a blistering pace.