Lightning fast data retrieval with YQL

Some people at Yahoo are obsessed with performance and speed of web services. For example, Stoyan Stefanov--who took it onto himself to write a performance article a day until Christmas in his performance advent calendar. Over instant messenger we got to chatting about using YQL as a proxy to retrieve data from the web faster. When he asked me to prove it, I put together a small guest post on how to use YQL to retrieve five RSS feeds much faster than with any other technology.

The results are pretty conclusive. By taking advantage of YQL's fast connectivity to the web and all the caching magic that goes on, you can cut the retrieval time for five RSS feeds from over 4 seconds using simple cURL to 0.08 seconds using YQL:

Retrieving five RSS feeds and converting it on the server with YQL execute by  you.

The speed of YQL is one thing. Using YQL execute to not only pull the data, but also convert it server-side with JavaScript, we can consume and aggregate RSS feeds in a very simple fashion. As explained in the blog post about the multi-RSS open table you can now use YQL to get RSS back as HTML lists:

use "http://github.com/yql/yql-tables/raw/master/data/rss.multi.list.xml" as mrss;
select * from mrss where feeds="
'http://search.twitter.com/search.rss?q=performance',
'http://search.twitter.com/search.rss?q=webservice',
'http://feeds.delicious.com/v2/rss/tag/performance?count=15'
" and html="true" and compact="true"

Try it out in the YQL console or see the XML output.

Once you've copied your URL from the console using this table is pretty easy. You can use PHP if you want to support everybody:

$url= "http://query.yahooapis.com/v1/public/yql?q=use%20%22http%3A%2F%2Fgithub.com%2Fyql%2Fyql-tables%2Fraw%2Fmaster%2Fdata%2Frss.multi.list.xml%22%20as%20mrss%3B%20select%20*%20from%20mrss%20where%20feeds%3D%22'http%3A%2F%2Fsearch.twitter.com%2Fsearch.rss%3Fq%3Dperformance%26rpp%3D5'%2C'http%3A%2F%2Fsearch.twitter.com%2Fsearch.rss%3Fq%3Dwebservice%26rpp%3D5'%2C'http%3A%2F%2Ffeeds.delicious.com%2Fv2%2Frss%2Ftag%2Fperformance%3Fcount%3D5'%22%20and%20html%3D%22true%22%20and%20compact%3D%22true%22&format=xml&diagnostics=false&callback=seedfeeds";
$content = get($url);
$content = preg_replace("/.*\[\"|\"\].*/","",$content);
echo stripslashes($content);
function get($url){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec($ch);
curl_close($ch);
return $output;
}

And as JSON-P-X is a string inside a JSON-P object you can immediately use it in JavaScript via innerHTML.

<div id="feeds">Loading...</div>
<script type="text/javascript">
function seedfeeds(o){
var f = document.getElementById('feeds');
f.innerHTML = o.results;
}
</script>
<script type="text/javascript" src="http://query.yahooapis.com/v1/public/yql?q=use%20%22http%3A%2F%2Fgithub.com%2Fyql%2Fyql-tables%2Fraw%2Fmaster%2Fdata%2Frss.multi.list.xml%22%20as%20mrss%3B%20select%20*%20from%20mrss%20where%20feeds%3D%22'http%3A%2F%2Fsearch.twitter.com%2Fsearch.rss%3Fq%3Dperformance%26rpp%3D5'%2C'http%3A%2F%2Fsearch.twitter.com%2Fsearch.rss%3Fq%3Dwebservice%26rpp%3D5'%2C'http%3A%2F%2Ffeeds.delicious.com%2Fv2%2Frss%2Ftag%2Fperformance%3Fcount%3D5'%22%20and%20html%3D%22true%22%20and%20compact%3D%22true%22&format=xml&diagnostics=false&callback=seedfeeds"</script>

Speed and ease of use - that's YQL for you.