YQL Tutorials and Code Examples

Introduction

This chapter provides the following:

Prerequisites

  • The Two-Minute Tutorial
  • Have a Web server that can be accessed over the Internet.
  • Be familiar with the following technologies:
    • SQL
    • HTML/XHTML
    • JavaScript
    • Web feeds

YQL Tutorials

First YQL Applications

Time Required: 10 min.

Introduction

This tutorial shows how to create a simple Web application that uses YQL to fetch an RSS feed. The call to YQL is made within an HTML script tag, and the returned JSON response is parsed with JavaScript.

Setting Up the Example

  1. From your Web server, create a new file called yql_news_app.html.

  2. Copy the HTML below into yql_news_app.html. The src attribute in the second script tag is given an empty value intentionally. We will assign a YQL statement to src later.

    <html>
      <head><title>YQL and RSS: Yahoo Top News Stories</title>
      <style type='text/css'>
        #results{ width: 40%; margin-left: 30%; border: 1px solid gray; padding: 5px; height: 200px; overflow: auto; }
      </style>
      <script type='text/javascript'>
        // Parses returned response and extracts
        // the title, links, and text of each news story.
        function top_stories(o){
          var items = o.query.results.item;
          var output = '';
          var no_items=items.length;
          for(var i=0;i<no_items;i++){
            var title = items[i].title;
            var link = items[i].link;
            var desc = items[i].description;
            output += "<h3><a href='" + link + "'>"+title+"</a></h3>" + desc + "<hr/>";
          }
          // Place news stories in div tag
          document.getElementById('results').innerHTML = output;
        }
        </script>
      </head>
      <body>
        <!-- Div tag for stories results -->
        <div id='results'></div>
        <!-- The YQL statment will be assigned to src. -->
        <script src=''></script>
      </body>
    </html>
    
  3. Run the example query get rss feed from yahoo top stories in the YQL Console. Select the JSON radio button and click TEST. You should see the returned JSON response on the Formatted View tab.

  4. In the YQL statement, replace the word “title” with an asterisk “*”. Click TEST. The asterisk will return all of the fields (rows) of the response.

  5. From the returned JSON response, find the results object. Notice that within the results object, there is the array item containing objects with the title, link, and description of each news article.

  6. Click Copy URL. From yql_news_app.html, paste the URL into the src attribute of the second script tag as seen below:

    <body>
      <div id='results'></div>
      <script src='http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20rss%20where%20url%3D%22http%3A%2F%2Frss.news.yahoo.com%2Frss%2Ftopstories%22&format=json&callback=cbfunc'>
      </script>
    </body>
    

    Notice that the YQL statement has been URL-encoded and the format parameter specifies that the response be in JSON.

  7. At the end of the URL, replace the callback value ‘cbfunc’ with ‘top_stories’. The new callback function top_stories will be called as soon as YQL returns the response and have access to the returned JSON.

  8. Point your browser to yql_news_app.html. Your YQL News Application should resemble the figure below.

    Screenshot of the YQL News Application
  9. (Optional) Use almost the exact same code to create an application for finding sushi restaurants in San Francisco.

    Replace the YQL statement for the Yahoo Top Stories in your YQL News Application with the example query find sushi restaurants in san francisco. Be sure that the format is JSON and that the name of the callback function is correct.

  10. Make a few changes below to the function top_stories as seen below:

    function top_stories(o){
       var items = o.query.results.Result;
       var output = '';
       var no_items=items.length;
       for(var i=0;i<no_items;i++){
           var title = items[i].Title;
           var link = items[i].Url;
           var desc = items[i].Rating.LastReviewIntro;
           output += "<h3><a href='" + link + "'>"+title+"</a></h3>" + desc + "<hr/>";
       }
       document.getElementById('results').innerHTML = output;
    }
    
  11. The YQL News Application is now the YQL Sushi Restaurant Finder Application.

    Screenshot of the YQL Sushi Restaurant Finder Application

Source Code

YQL Top News Application
<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd" https="true">
    <meta>
        <author>Paul Donnelly</author>
        <documentationURL>http://developer.netflix.com/docs/REST_API_Reference#0_52696</documentationURL>
    </meta>
      <bindings>
        <select itemPath="" produces="XML" >
          <urls>
            <url>http://api-public.netflix.com/catalog/titles/</url>
          </urls>
         <paging model="offset">
           <start id="start_index" default="0" />
           <pagesize id="max_results" max="100" />
           <total default="20" />
         </paging>
          <inputs>
                <key id="term" type="xs:string" paramType="query" required="true" />
                <key id="ck" type="xs:string" paramType="variable" required="true" />
                <key id="cs" type="xs:string" paramType="variable" required="true" />
          </inputs>
          <execute><![CDATA[
// Include the OAuth libraries from oauth.net
y.include("http://oauth.googlecode.com/svn/code/javascript/oauth.js");
y.include("http://oauth.googlecode.com/svn/code/javascript/sha1.js");

// Collect all the parameters
var encodedurl = request.url;
var accessor = { consumerSecret: cs, tokenSecret: ""};
var message = { action: encodedurl, method: "GET", parameters: [["oauth_consumer_key",ck],["oauth_version","1.0"]]};
OAuth.setTimestampAndNonce(message);

// Sign the request
OAuth.SignatureMethod.sign(message, accessor);

try {
   // get the content from service along with the OAuth header, and return the result back out
   response.object = request.contentType('application/xml').header("Authorization", OAuth.getAuthorizationHeader("netflix.com", message.parameters)).get().response;
} catch(err) {
   response.object = {'result':'failure', 'error': err};
}
          ]]></execute>
        </select>
      </bindings>
</table>

<html>
  <head><title>YQL and RSS: Yahoo Top News Stories</title>      <style type='text/css'>
      #results{ width: 40%; margin-left: 30%; border: 1px solid gray; padding: 5px; height: 200px; overflow: auto; }
    </style>
    <script type='text/javascript'>
      function top_stories(o){
        var items = o.query.results.item;
        var output = '';
        var no_items=items.length;
        for(var i=0;i<no_items;i++){
          var title = items[i].title;
          var link = items[i].link;
          var desc = items[i].description;
          output += "<h3><a href='" + link + "'>"+title+"</a></h3>" + desc + "<hr/>";
        }
        document.getElementById('results').innerHTML = output;
      }
    </script>
  </head>
  <body>
    <div id='results'></div>
    <script src='http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20rss%20where%20url%3D%22http%3A%2F%2Frss.news.yahoo.com%2Frss%2Ftopstories%22&format=json&diagnostics=false&callback=top_stories'></script>
  </body>
</html>

Creating YQL Open Data Tables

Time Required: 15 min.

Introduction

This tutorial shows you how create a YQL Open Data Table for the Bay Area Region Transit (BART) Real-Time Information API. You will then use the YQL console to run a YQL query using the Open Data Table that you created.

What Are Open Data Tables?

YQL already contains an extensive list of built-in tables for you to use that cover a wide range of Yahoo Web services and access to off-network data. Developers can now add to this list of built-in tables by creating their own Open Data Tables. YQL allows you to create and use your own table definitions, enabling YQL to bind to any data source through the SQL-like syntax and fetch data. Once an Open Data Table is created, anyone can use these definitions in YQL.

By creating an Open Data Table for the feed, you can then use YQL to make queries on the data from the feed and mash up other data from existing YQL tables with the social data. For more information about Open Data Tables, see Using YQL Open Data Tables.

Creating and Using Your Open Data Table

  1. Copy bart.xml to your Web server. The file bart.xml defines the Open Data Table, which allows the YQL Web service to access data from the BART Real-Team Information API. For a more detailed explanation, see A Closer Look at the Code.

  2. Go to the YQL Console.

  3. From the YQL console, type the YQL statement below in the “Your YQL Statement” text area and click the button TEST. Be sure to replace “your_domain_name” with the domain name of your Web site.

    USE "http://your_domain_name/bart.xml" AS bart_table; SELECT * FROM bart_table;
    
  4. From the “Formatted View” tab, you should see results similar to those below:

    <results>
      <station>
        <name>Lake Merritt</name>
        <abbr>LAKE</abbr>
        <etd>
          <destination>Daly City</destination>
          <abbreviation>DALY</abbreviation>
          <estimate>
            <minutes>2</minutes>
            <platform>2</platform>
            <direction>North</direction>
            <length>8</length>
            <color>BLUE</color>
            <hexcolor>#0099cc</hexcolor>
            <bikeflag>1</bikeflag>
          </estimate>
          ...
         </etd>
         <etd>
           ...
         </etd>
         ...
       </station>
       <station>
         ...
       </station>
       ...
    </results>
    
  5. Try a few more YQL statements using the Open Data Tables. Below are a few sample YQL statements.

    • This statement will get you the station name and ETAs for any train going to the San Francisco International Airport.

      USE "http://your_domain_name/bart.xml" AS bart_table; SELECT name, etd.destination, etd.estimate FROM bart_table WHERE name LIKE "%24th%"
      

A Closer Look at the Code

This section will examine in greater detail at the Open Data Table and the YQL queries that are used in this tutorial.

This section will examine in greater detail at the Open Data Table and the YQL queries that are used in this tutorial.

Open Data Table

The Open Data Table must conform to the XML schema table.xsd, which is given as the XML namespace attribute for the element table. The element meta has child elements that provide general information about the Open Data Table such as the author, the location of the documentation, and a sample query, as shown below.

<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd">
    <meta>
        <author>You</author>
        <documentationURL>None</documentationURL>
        <description>Uses the BART ETA feed.</description>
        <sampleQuery>SELECT * FROM {table} WHERE orig="12th"</sampleQuery>
        <sampleQuery>SELECT * FROM {table} WHERE orig="12th" where orig={orig}</sampleQuery>
        <sampleQuery>SELECT * FROM {table} WHERE orig="12th" where orig={orig} and key={key}</sampleQuery>
    </meta>

The element bindings allows YQL to map the data source so that information can be queried. The element select defines what repeating element of the XML from the data source will act as a row of a table with the attribute itemPath, which is <root><station>...</station> (given as root.element) in the code below. A good example of a repeating element in XML would be the element entry in an Atom 1.0 feed. We also use the key element to append query parameters to the specified URL. The key elements take advantage of the default property, so you don’t have to specify these keys in the YQL statement, but you still have the flexibility to choose the originating location with the key orig and use your own BART API key.

<bindings>
   <select itemPath="root.station" produces="XML">
     <urls>
        <url>http://api.bart.gov/api/etd.aspx</url>
     </urls>
     <inputs>
       <key id="orig" as="orig" type="xs:string" paramType="query" required="false" default="ALL" />
       <key id="key" as="key" type="xs:string" paramType="query" required="true" default="MW9S-E7SL-26DU-VV8V" />
       <key id="cmd" as="cmd" type="xs:string" paramType="query" required="false" default="etd" />
     </inputs>
   </select>
 </bindings>

YQL Query

You invoke an Open Data Table with the verbs USE and AS. YQL fetches the Open Data Table defintion pointed to by the verb USE and then the verb AS creates an alias to that definition.

USE "http://your_domain_name/bart.xml" AS bart_table;

The alias bart_table can now be accessed with a YQL statement. The simple YQL query below returns all the rows from the BART feed.

SELECT * FROM bart_table;

Source Code

<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd">
  <meta>
    <author>You</author>
    <documentationURL>None</documentationURL>
    <description>Uses the BART ETA feed.</description>
    <sampleQuery>SELECT * FROM {table} WHERE orig="12th"</sampleQuery>
    <sampleQuery>SELECT * FROM {table} WHERE orig="12th" where orig={orig}</sampleQuery>
    <sampleQuery>SELECT * FROM {table} WHERE orig="12th" where orig={orig} and key={key}</sampleQuery>
  </meta>
  <bindings>
    <select itemPath="root.station" produces="XML">
      <urls>
        <url>http://api.bart.gov/api/etd.aspx</url>
      </urls>
      <inputs>
        <key id="orig" as="orig" type="xs:string" paramType="query" required="false" default="ALL" />
        <key id="key" as="key" type="xs:string" paramType="query" required="true" default="MW9S-E7SL-26DU-VV8V" />
        <key id="cmd" as="cmd" type="xs:string" paramType="query" required="false" default="etd" />
      </inputs>
    </select>
  </bindings>
</table>

Executable JavaScript in Open Data Tables

Time Required: 15-20 min.

Introduction

This tutorial shows how to include JavaScript in an Open Data Table definition that is executed by the YQL Web service. The JavaScript in the Open Data Table for this tutorial includes external libraries and uses OAuth to make an authorized call to the Netflix API. With the Netflix Open Data Table, you can then run YQL statements from the YQL console or create an application that searches and displays movies from the Netflix catalog.

Including JavaScript in an Open Data Table definition allows you to do the following:

  • Access external libraries
  • Make requests to Web services
  • Use E4X to structure the response in XML
  • Dynamically control the response with conditional logic
  • Perform Web authorization

Prerequisites

Executable JavaScript

JavaScript is placed in the element execute of an Open Data Table definition. When a SELECT statement calls an Open Data Table definition having the element execute, YQL executes the JavaScript on the backend and returns the response generated by the JavaScript. Open Data Table definitions having the element execute must return a response formed by the JavaScript.

In contrast, Open Data Tables not having the element execute make GET requests to the URI defined in the element url and return the response from that URI resource. We’ll now look at a couple of examples of Open Data Tables with and without the element execute to illustrate the difference.

Example of Open Data Table Definition without JavaScript

The Open Data Table definition below does not include the element execute, so YQL makes a GET request to the URI defined in the element url: http://www.bart.gov/dev/eta/bart_eta.xml

<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd">
  <meta>
    <author>You</author>
    <documentationURL>None</documentationURL>
    <description>Uses the BART ETA feed.</description>
    <sampleQuery>SELECT * FROM {table} WHERE abbr="12th"</sampleQuery>
  </meta>
  <bindings>
    <select itemPath="root.station" produces="XML">
      <urls>
        <url>http://www.bart.gov/dev/eta/bart_eta.xml</url>
      </urls>
    </select>
  </bindings>
</table>

Example of Open Data Table Definition with JavaScript

Because of the presence of the element execute, YQL executes the JavaScript and returns the response assigned to response.object, which will be discussed in Creating the Response.

<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd">
  <meta>
    <sampleQuery>select * from {table} where a='cat' and b='dog';</sampleQuery>
  </meta>
  <bindings>
    <select itemPath="" produces="XML">
      <urls>
        <url>http://fake.url/{a}</url>
      </urls>
      <inputs>
        <key id='a' type='xs:string' paramType='path' required="true" />
        <key id='b' type='xs:string' paramType='variable' required="true" />
      </inputs>
      <execute><![CDATA[
         // Your javascript goes here.
         // We will run it on our servers
         response.object = <item>
                             <url>{request.url}</url>
                             <a>{a}</a>
                             <b>{b}</b>
                           </item>;
      ]]></execute>
    </select>
  </bindings>
</table>

A Closer Look at the Code

JavaScript in Open Data Tables definitions have access to the three global objects y, request, and response. From these global objects, methods can be called to include external libraries, perform 2-legged authorization , make requests to Web services, run YQL queries, convert data structures, such as XML to JSON, and more. We’ll be looking at the methods used in this tutorial that allow you to include external libraries, make requests to Web services, and extract the results from a returned response.

To read more about the JavaScript global objects and the available methods, see the JavaScript Objects and Methods Reference.

Including External Libraries

The global object y includes the method include that allows you to reference external JavaScript libraries from an Open Data Table definition. Because calls to the Netflix API must be authorized with OAuth, you include external libraries to handle the authorization and sign the request as shown in the code below:

<execute>
    y.include("http://oauth.googlecode.com/svn/code/javascript/oauth.js");
    y.include("http://oauth.googlecode.com/svn/code/javascript/sha1.js");
    ...
</execute>

Making Requests

To make a request to a Web service, you use the method get from either the global object request or from an instance of request; the instance is created by calling the method rest from the global object y. Before looking at the code used in the Open Data Table definition for this tutorial, let’s look at a couple of simple examples of using both the global object and an instance of request.

This example creates an instance of request by passing the URI resource to the method rest. From the instance, the method get can make the request to the Web service.

// Create an instance of the request object
var request_instance = y.rest("http://some_web_service");

// Make request to web service with HTTP method 'GET'
var returned_response = request_instance.get();

You can also call the method get directly from the global object request. The element url holds the URI to the Web service, which is stored in request.url.

...
<select itemPath="" produces="XML">
  <urls>
    <url>http://some_web_service_or_feed</url>
  </urls>
  <execute><![CDDATA[
    // request.url == 'http://some_web_service_or_feed
    // This is the same as 'y.rest(request.url).get();'
    var returned_response = request.get();
    ...
  ]]></execute>
</select>
...

You can also pass in parameters and define the content returned when making a request to a Web service, which is needed to make authorized requests, such as the request to the Netflix API in this tutorial.

To get the response from the Netflix API, the content type must be defined as 'application/xml', and the OAuth authorization header must be passed. The code snippet below also uses the objects response and response.object to extract results from the returned response and create the response that YQL returns. We’ll examine the object response in Getting and Creating a Response.

// The content type is defined and the OAuth header
// is passed to get a response from the Netflix API
response.object = request.contentType('application/xml').header("Authorization", OAuth.getAuthorizationHeader("netflix.com", message.parameters)).get().response;

Getting and Creating the Response

Extracting Data From the Returned Response

To get the data (results) from the returned response, you reference the object response. From this earlier code example, you can see the dot operator being used to reference the results of the returned response.

// Create an instance of the request object
var request_obj = y.rest("http://some_web_service");

// Make request to web service with HTTP method 'GET'
// and get the results by referencing 'response'
var response_results = request_obj.get().response;

We also saw this example of using the response object to get the data from the Netflix Web service.

// The object response lets you extract the
// results from the returned response
response.object = request.contentType('application/xml').header("Authorization", OAuth.getAuthorizationHeader("netflix.com", message.parameters)).get().response;

Creating the Response

When including JavaScript in an Open Data Table, the response from the YQL Web service is determined by the data that is assigned to the object response.object. You can use the returned value from a Web service or create your own with JSON or E4X.

In this code snippet, an XML literal is created with E4X and then assigned to response.object. To interpolate the variable into the XML literal, enclose the variable in braces.

// Use E4X to create XML literals with
// interpolated variable and assign to
// response.object
var error = "Failed to get response";
response.object = <error><message>{error}</message></error>;

The code from the tutorial assigns the returned XML response to response.object as seen below. If there is an error, a JSON object holding error message is returned.

try {
    // get the content from service along with the OAuth header, and return the result back out
    response.object = request.contentType('application/xml').header("Authorization", OAuth.getAuthorizationHeader("netflix.com", message.parameters)).get().response;
} catch(err) {
    response.object = {'result':'failure', 'error': err};
}

Paging Results

The element paging lets you have more control over the results returned from the YQL query. We’ll look in detail at the paging used in the Netflix Open Data Table that is shown below. For more information about paging, see the Open Data Tables Reference and Paging Results.

<paging model="offset">
    <start id="start_index" default="0" />
    <pagesize id="max_results" max="100" />
    <total default="20" />
</paging>

The value ‘offset’ for the attribute model states that the YQL query can state an offset from the start of the results. The ability to get an offset from a result set depends on the source of the data, which in the tutorial is the Netflix API. Be sure to verify that your data source allows retrieving data from an offset when you create future Open Data Tables.

<paging model="offset">

The default offset is set by the attribute default in the element start, which is 0 or no offset in this example.

<start id="start_index" default="0" />

The maximum number of results that can be returned by a YQL query on this table is 100, which is defined by the attribute max.

<pagesize id="max_results" max="100" />

The attribute default in the element total states the default number of results returned for each page.

<total default="20" />

Setting Up This Example

To set up the Netflix Open Data Table and run YQL queries:

  1. Copy netflix_table.xml to your Web server. The file netflix_table.xml defines the Open Data Table, which allows the YQL Web service to access data from the NetFlix API. For a more detailed explanation, see A Closer Look at the Code.

  2. If you do not have Netflix API keys, register for a Netflix developer account and apply for an API key.

  3. Go to the YQL Console. Make sure that the URL of the YQL Console begins with https.

  4. Select the first 20 results (see ref:Paging Results <js_e4x-paging_results>) for movies that match the key word “Rocky”. Be sure to use your Consumer Key and Consumer Secret from your Netflix account for the values for ck and cs.

    USE "http://your_domain/netflix.xml" AS netflix_table; SELECT * FROM netflix_table
    WHERE term="rocky" AND ck="your_consumer_key" AND cs="your_consumer_secret"
    
  5. In this YQL statement, you ask for the first 10 results for movies, and the results will include the film title, the URL of the box art image, and the average rating.

    USE "http://yourdomain/netflix.xml" AS netflix_table; SELECT catalog_title.title,
    catalog_title.box_art.medium, catalog_title.average_rating FROM netflix_table(10)
    WHERE term="rocky" AND ck="your_consumer_key" AND cs="your_consumer_secret"
    
  6. You can also specify an offset for the returned results. The YQL statements below returns the five titles and ratings for Star Trek movies with average ratings over 3.0. The results are sorted by the rating in descending order, and the offset is five.

    USE "http://yourdomain/netflix.xml" AS netflix_table; SELECT catalog_title.title.regular,
    catalog_title.average_rating FROM netflix_table(5,5) WHERE term="star trek" AND
    ck="your_consumer_key" AND cs="your_consumer_secret" AND catalog_title.average_rating > "3.0" | sort(field="catalog_title.average_rating") | reverse()
    

Source Code

Netflix Open Data Table

<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd" https="true">
    <meta>
        <author>Paul Donnelly</author>
        <documentationURL>http://developer.netflix.com/docs/REST_API_Reference#0_52696</documentationURL>
    </meta>
      <bindings>
        <select itemPath="" produces="XML" >
          <urls>
            <url>http://api-public.netflix.com/catalog/titles/</url>
          </urls>
         <paging model="offset">
           <start id="start_index" default="0" />
           <pagesize id="max_results" max="100" />
           <total default="20" />
         </paging>
          <inputs>
                <key id="term" type="xs:string" paramType="query" required="true" />
                <key id="ck" type="xs:string" paramType="variable" required="true" />
                <key id="cs" type="xs:string" paramType="variable" required="true" />
          </inputs>
          <execute><![CDATA[
// Include the OAuth libraries from oauth.net
y.include("http://oauth.googlecode.com/svn/code/javascript/oauth.js");
y.include("http://oauth.googlecode.com/svn/code/javascript/sha1.js");

// Collect all the parameters
var encodedurl = request.url;
var accessor = { consumerSecret: cs, tokenSecret: ""};
var message = { action: encodedurl, method: "GET", parameters: [["oauth_consumer_key",ck],["oauth_version","1.0"]]};
OAuth.setTimestampAndNonce(message);

// Sign the request
OAuth.SignatureMethod.sign(message, accessor);

try {
   // get the content from service along with the OAuth header, and return the result back out
   response.object = request.contentType('application/xml').header("Authorization", OAuth.getAuthorizationHeader("netflix.com", message.parameters)).get().response;
} catch(err) {
   response.object = {'result':'failure', 'error': err};
}
          ]]></execute>
        </select>
      </bindings>
</table>

YQL Code Examples

Prerequisites

Making YQL Queries With JavaScript

Summary

This code example shows you how to make YQL queries with JavaScript and the OpenSocial function makeRequest. The YQL query in the code example will get data from the GeoPlanet API based on the user’s input.

OpenSocial Method: makeRequest

By using the OpenSocial method gadgets.io.makeRequest, you can make calls to a Web service with JavaScript without using a crossdomain.xml file. The calls in this example use 2-legged OAuth authorization.

Before calling the function makeRequest, define the base URI of the YQL Web service, the YQL query, and a callback function to handle the response, as shown below:

var BASE_URI = 'http://query.yahooapis.com/v1/yql';
// Base URI for Web service
var yql_base_uri = "http://query.yahooapis.com/v1/yql";

// Create a variable to make results available
// in the global namespace
var yql_results = "";

// Create a YQL query to get geo data for the
// San Francisco International Airport
var yql_query = "SELECT * from geo.places WHERE text='SFO'";

// Callback function for handling response data
function handler(rsp) {
  if(rsp.data){
    yql_results = rsp.data;
  }
}

To make a call to the YQL Web service, you create a URL-encoded query string with the YQL query and the requested format type of the response and append this query string to the base URI of the YQL Web service. The code snippet below shows the base URI with the appended query string for the YQL Web service. Note that the spaces have been URL-encoded.

<code class="literal">http://query.yahooapis.com/v1/public/yql?q=SELECT%20*%20FROM%20geo.places%20WHERE%20text%3D%22SFO%22&format=json</code>

This code example has the JavaScript function toQueryString that creates the query string for the call to the YQL Web service as seen below:

// This utility function creates the query string
// to be appended to the base URI of the YQL Web
// service.
function toQueryString(obj) {
  var parts = [];
  for(var each in obj) if (obj.hasOwnProperty(each)) {
    parts.push(encodeURIComponent(each) + '=' + encodeURIComponent(obj[each]));
  }
  return parts.join('&');
};

By using a closure in the code below, the variable query, which holds the YQL query, is available to makeRequest. The function toQueryString then builds the query string, which is appended to the base URI by makeRequest before making the call to the YQL Web service.

// Store the anonymous function that wraps
// the OpenSocial function makeRequest
var runQuery = function(ws_base_uri,query, handler) {
  gadgets.io.makeRequest(ws_base_uri, handler, {
    METHOD: 'POST',
    POST_DATA: toQueryString({q: query, format: 'json'}),
    CONTENT_TYPE: 'JSON',
    AUTHORIZATION: 'OAuth'
  });
};

In the code snippet, the function object runQuery is passed the parameters for makeRequest to call the YQL Web service. The handler returns the response data (JSON object) to the OpenSocial function stringify to be converted to a string so it can be displayed in a div tag.

<div id="results"></div>
<script>
  // Call YQL Web service and use YQL query
  // to get results from the GEO
  runQuery(yql_base_uri,yql_query,handler);

  // Use stringify function from OpenSocial library
  // to convert JSON to string and display the string
  // in the div with the id 'results'
  document.getElementById('results').innerHTML = gadgets.json.stringify(yql_results.data);
</script>

Source Code

<form id="geosearch">
<p><label for="query">Enter Location:</label>
<input id="query" type="text"/></p>
<p><input type="submit" value="Make Query"/></p>
</form>
<div id="results"></div>

<script>
/*
  This example shows how to use YQL to
  make queries to the GEO Web service.
  The call to the YQL Web service uses
  2-legged OAuth and is made with OpenSocial
  functions.
*/
function makeQuery(e){
  e.preventDefault(); // do not send off form
  var container = document.getElementById('results');
  var location = document.getElementById('query').value || 'SFO';
  var content = '';

  var BASE_URI = 'http://query.yahooapis.com/v1/yql';

  // function calling the opensocial makerequest method
  function runQuery(query, handler) {
    gadgets.io.makeRequest(BASE_URI, handler, {
        METHOD: 'POST',
        POST_DATA: toQueryString({q: query, format: 'json'}),
        CONTENT_TYPE: 'JSON',
        AUTHORIZATION: 'OAuth'
    });
  };

  // Tool function to create a request string
  function toQueryString(obj) {
    var parts = [];
    for(var each in obj) if (obj.hasOwnProperty(each)) {
      parts.push(encodeURIComponent(each) + '=' +
                 encodeURIComponent(obj[each]));
    }
    return parts.join('&');
  };

  // Run YQL query to GeoPlanet API and extract data from response
  runQuery('select * from geo.places where text="' + location + '"',
    function(rsp) {
      if(rsp.data){
        var place = rsp.data.query.results.place;
        if(place[0]){
          place = place[0];
        }
        var name      = place.name || 'Unknown';
        var country   = place.country.content || place[0].country.content ||
                        'Unknown';
        var latitude  = place.centroid.latitude || 'Unknown';
        var longitude = place.centroid.longitude || 'Unknown';
        var city      = place.locality1.content || 'Unknown';
        var state     = place.admin1.content || 'Unknown';
        var county    = place.admin2.content || 'Unknown';
        var zip       = place.postal ? place.postal.content : 'Unknown';

        content = '<ul><li><strong>Place Name: </strong>' + name + '</li>'+
        '<li><strong>City/Town: </strong>' + city + '</li>' +
        '<li><strong>County/District: </strong>' + county + '</li>' +
        '<li><strong>State/Province: </strong>' + state + '</li>' +
        '<li><strong>Zipcode: </strong>' + zip + '</li>' +
        '<li><strong>Country: </strong>' + country + '</li>' +
        '<li><strong>Latitude: </strong>' + latitude + '</li>' +
        '<li><strong>Longitude: </strong>' + longitude + '</li></ul>';
        container.innerHTML = content;
      }
      else {
        container.innerHTML = gadgets.json.stringify(rsp);
      }
  });
}
// Create an event handler for submitting the form
var form = document.getElementById('geosearch');
form.addEventListener('submit',makeQuery,false);
</script>

Making YQL Queries With PHP

Summary

This example, yql_php.php, is a simple application that uses cURL to make YQL calls to the Upcoming API.

To understand the following material, you should already be familiar with the topics covered in the The Two-Minute Tutorial.

Building the YQL URL

To build the YQL URL, append the YQL query to the base URL of the YQL Web service.

  1. First, assign the base URL of the YQL Web service to a variable.

    $yql_base_url = "http://query.yahooapis.com/v1/public/yql";
    
  2. Now create the YQL query to the Upcoming API and append it to the YQL base URL. YQL queries are passed as query strings, so they must be URL encoded. The query string must begin with q=.

    $yql_query = "select * from upcoming.events where location='San Francisco' and search_text='dance'";
    
    $yql_query_url = $yql_base_url . "?q=" . urlencode($yql_query);
    
  3. The YQL Web service returns XML as the default format. This code example requests JSON by appending the name-value pair format=json to the query string as shown.

    $yql_query_url .= "&format=json";
    
  4. Run the query in the YQL console and look at both the URL in The REST query window and the JSON response in the Formatted View window.

Calling the YQL Web Service with cURL

Calling the YQL Web service with cURL only requires three lines of code. After initializing the call by passing the curl the YQL URL, you request a response with curl_setopt and then execute the call.

$session = curl_init($yql_query_url);
curl_setopt($session, CURLOPT_RETURNTRANSFER,true);
$json = curl_exec($session);

To make the response easier to handle, convert the JSON response to a PHP object to easily access data.

$phpObj =  json_decode($json);

Parsing the Response

The YQL Web service will always return results wrapped in the query field with meta data. If the YQL query returns data, the data is wrapped in the results field, which you can see by running this query.

If no data is returned, the results element in the XML response is empty or the results field in JSON is null. Therefore, before parsing the response returned by YQL, your code should always check the results field as shown below:

if(!is_null($phpObj->query->results)){
  // Safe to parse data
}

The structure of the data within the results field that is returned by YQL is different for each API. The repeated field (like a row in an SQL table) in the returned response in this code example is the event field. The code below extracts and displays data from each event:

if(!is_null($phpObj->query->results)){
  foreach($phpObj->query->results->event as $event){
    $events .= "<div><h2>" . $event->name . "</h2><p>";
    $events .= html_entity_decode(wordwrap($event->description, 80, "<br/>"));
    $events .="</p><br/>$event->venue_name<br/>$event->venue_address<br/>";
    $events .="$event->venue_city, $event->venue_state_name";
    $events .="<p><a href=$event->ticket_url>Buy Tickets</a></p></div>";
  }
}

Source Code

<h2>Using YQL to Access the Upcoming API</h2>
<form name='upcoming_form'>
Location: <input name='location' id='location' type='text' size='20'/><br/>
Event: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<input name='event' id='event' type='text' size='20'/><br/>
<p><button id='find_event'>Find Event</button></p>
</form>

<script>
  // Attach event handler to button
  document.getElementById("find_event").addEventListener("click",find_event,false);
  // Get user input and submit form
  function find_event(){
    document.upcoming_form.event.value = document.getElementById('event').value || "music";
    document.upcoming_form.location.value = document.getElementById('location').value || "San Francisco";
    document.upcoming_form.submit();
  }
</script>
<?php
  $BASE_URL = "https://query.yahooapis.com/v1/public/yql";

  if(isset($_GET['event']) && isset($_GET['location'])){
    $location = $_GET['location'];
    $query = $_GET['event'];
    $events="";

    // Form YQL query and build URI to YQL Web service
    $yql_query = "select * from upcoming.events where location='$location' and search_text='$query'";
    $yql_query_url = $BASE_URL . "?q=" . urlencode($yql_query) . "&format=json";

    // Make call with cURL
    $session = curl_init($yql_query_url);
    curl_setopt($session, CURLOPT_RETURNTRANSFER,true);
    $json = curl_exec($session);
    // Convert JSON to PHP object
    $phpObj =  json_decode($json);

    // Confirm that results were returned before parsing
    if(!is_null($phpObj->query->results)){
      // Parse results and extract data to display
      foreach($phpObj->query->results->event as $event){
        $events .= "<div><h2>" . $event->name . "</h2><p>";
        $events .= html_entity_decode(wordwrap($event->description, 80, "<br/>"));
        $events .="</p><br/>$event->venue_name<br/>$event->venue_address<br/>";
        $events .="$event->venue_city, $event->venue_state_name";
        $events .="<p><a href=$event->ticket_url>Buy Tickets</a></p></div>";
      }
    }
    // No results were returned
    if(empty($events)){
      $events = "Sorry, no events matching $query in $location";
    }
    // Display results and unset the global array $_GET
    echo $events;
    unset($_GET);
  }
?>

Data Scraping With YQL

Summary

This example, yql_html_scraper.html, uses YQL to scrape HTML from Yahoo News and then creates customized output. The YQL module for YUI is used to call the YQL Web service.

Using the YQL Module for YUI

You can can download the YQL module for YUI from GitHub or include it in your Web page like any other JavaScript file. The steps below will show you how to use the YQL module to call the YQL Web service.

  1. Include the YUI library, which includes the YQL module, in your Web page.

    <script src="http://yui.yahooapis.com/3.8.0/build/yui/yui-min.js"></script>
    
  2. Create a YUI instance to load the Node and YQL modules.

    YUI().use('node', 'event', 'yql, function(Y) {
      // The code for manipulating DOM, handling events,
      // calling the YQL Web service, and parsing
      // the response goes here.
    })
    
  3. Using the html table, create a YQL query to get data from Yahoo News. See Using XPath in YQL Statements for details about the YQL query.

    var yql_query = "select * from html where url='" + news_url + story + "'";
    yql_query += " and xpath='//div[@class=\"content\"]//div[@class=\"txt\"]/p'";
    
  4. Instantiate a YQL object by passing the YQL query and a callback function to Y.YQL.

    Y.YQL(yql_query, function(response) {
        ...
      }
    );
    
  5. Make sure that results were returned and then parse those results.

    var stories = "<div><ul>";
    if(response.query.results){
      var no_stories = response.query.results.p.length;         var paras = response.query.results.p;
      paras.forEach(function(node,index) {
        if (node.hasOwnProperty('a') && node.hasOwnProperty('content')) {
          stories += "<li><a href='" + news_url + node.a.href + "' title='" + node.a.title + "'>" + node.content + "</a></li>";
        }
      });
    } else{
      stories += "<li>Sorry, could not find any headlines for the category " + story + ". Please try another one.</li>";
    }
    stories += "</ul></div>";
    

    Because each a element is a seen as row of data by YQL, you need to iterate through each a element to access the href and content child elements. Run the query in the YQL Console to examine the data structure or the returned response in greater detail.

Using XPath in YQL Statements

The YQL statement below is used in this code example to access the links for stories from Yahoo News. This section will examine the data source for the YQL statement and the XPath expression that extracts the links.

select * from html where url='http://news.yahoo.com/{category}' and xpath='//div[@class="content"]//div[@class="txt"]/p'

The YQL query in this example allows you to access HTML using the html table. YQL also has tables for accessing other data formats, such as XML, CSV, JSON, RSS, XST, etc.

select * from html

The key url determines the data source for the YQL query. The data source in this example is World News page on Yahoo News.

url='http://news.yahoo.com/world'

The XPath statement below extracts the <p> elements from <div> element with the class txt. Each paragraph (p element) is returned as a row (repeated element in XML).

xpath='//div[@class="content"]//div[@class="txt"]/p'

For more information about XPath, see WC3 XPath Language.

Source Code

<b>Stories: </b> <input type='text' size='15' id='story' value='world'/><br/><br/>
<button id='get_stories'>Get Stories</button>
<div id='results'></div>
<script src="https://yui-s.yahooapis.com/3.8.0/build/yui/yui-min.js"></script>
<script>
// Calls YQL Web service, parses results, and outputs results
YUI().use('node', 'event', 'yql', function(Y) {
  Y.one("#get_stories").on('click',function() {
    var stories = "<div><ul>";
    var story = Y.one('#story').get('value') || 'world';
    var news_url = "http://news.yahoo.com/";
    var yql_query = "select * from html where url='" + news_url + story + "'";
    yql_query += " and xpath='//div[@class=\"content\"]//div[@class=\"txt\"]/p'";
    Y.YQL(yql_query, function(response) {
      if(response.query.results){
        var no_stories = response.query.results.p.length;
        var paras = response.query.results.p;
        paras.forEach(function(node,index) {
          if (node.hasOwnProperty('a') && node.hasOwnProperty('content')) {
            stories += "<li><a href='" + news_url + node.a.href + "' title='" + node.a.title + "'>" + node.content + "</a></li>";
          }
       });
      } else{
        stories += "Sorry, could not find any headlines for the category " + story + ". Please try another one.";
      }
      stories += "</ul></div>";
      Y.one('#results').append(stories);
      stories = "";
    });
  });
});
</script>

Sample Open Data Tables

The following table has links to examples of YQL Open Data Tables:

Sample Code Description
Hello World Table Allows you to search a fictional table in which “a” is the path and “b” is the term.
Yahoo Messenger Status Allows you to see the status of a Yahoo Messenger user.
OAuth Signed Request to Netflix Allows you to make a two-legged OAuth signed request to Netflix.
Request for a Flickr frob Returns the Flickr frob, which is analogous to the request token in OAuth.
CSS Selector for HTML Allows you to filter HTML using CSS selectors.
Bit.ly Shorten URL (INSERT) Shortens a long URL into a bit.ly link.

YQL Screencasts

A screencast is a multimedia demonstration of an application, a tool, or a service. Screencasts guide you through Yahoo APIs and related tools using audio and video.

The following screencasts are related to YQL:

Screencast Description Author/Source
Introducing YQL This is an introductory screencast to YQL using the YQL Console. Jatin Billimoria, YDN
YDN Screencast: YQL Discusses how to access public and private Web services, filter and join results, extract data from Web pages, and extend YQL to query additional data sources. Dan Theurer and Jonathan Trevor, YDN
YQL Execute This is an intermediate-level screencast that discusses Open Data Tables and the use of JavaScript to manipulate data within Open Data Tables. Sam Pullara and Nagesh Susarla, YDN
YQL Console Update This screencast discusses updates to the YQL Console, including the REST query box and query aliases. Paul Donnelly and Jatin Billimoria, YDN
YQL Query Builder This screencast discusses the query builder feature, which allows developers who are new to YQL to explore tables and create queries easily and quickly. Paul Donnelly and Jatin Billimoria, YDN

YQL Slideshows

The following slideshows discuss YQL and its usage:

Slideshow Description Author/Source
YQL - A Query Language for the Web Jonathan Trevor, YQL Lead, gives an overview of the YQL Web service and YQL language. Jonathan Trevor, Slideshare
Open Hack London - Introduction to YQL This narrated slideshow introduces YQL and uses code example to show its main features. Christian Heilmann, Slideshare
YQL, Flickr, OAuth, YAP This slideshow offers a step-by-step explanation of how to use YQL and OAuth together to get Flickr photos. Erik Eldridge, Slideshare
BayJax July 2009 - Browser MVC with YQL & YUI In this slideshow, you learn how to use YQL and YUI together to create widgets. YQL retrieves the data (the model layer), and YUI handles the presentation and control layers. Jonathan LeBlanc, Slideshare

Additional Tutorials and Code Examples

The following advanced tutorials and code examples show specialized uses for YQL:

Tutorial/Code Example Description Author/Source
Building a (re)search interface for Yahoo, Bing and Google with YQL Uses a single YQL statement with an Open Data Table to make queries to three search engines. Chris Heilmann, http://www.wait-till-i.com
Getting stock information with YQL and open data tables Turns Yahoo Finance into an API with executable JavaScript in an Open Data Table. YQL Team, http://www.yqlblog.net/blog
Tutorial: scraping and turning a website into a widget with YQL Scrapes a website with YQL and XPath to make a TV fun facts widget. Chris Heilmann, http://www.wait-till-i.com

Additional References and Resources

The following references and resources for YQL include libraries, handouts, blog posts, and demos:

Reference/Resource Description Author/Source
YQL Module for YUI JavaScript library for making queries to the YQL Web service. Dav Glass, http://yuilibrary.com
Python-YQL: client library for YQL Python library for making queries to the YQL Web service. Stuart Colville, http://muffinresearch.co.uk
Using YQL for Developers (PDF) Handout that briefly introduces YQL and serves as a cheat sheet. YDN, http://developer.yahoo.com
Creating Open Data Tables for YQL (PDF) Handout that introduces and provides examples of YQL Open Data Tables. YDN, http://developer.yahoo.com
YQL: INSERT INTO internet YDN blog post that discusses the YQL INSERT INTO statement. YQL examples demo links are also given. Jonathan Trevor, YDN
GooHooBi Search YQL-powered application that returns search results from Google, Yahoo, and Bing. Chris Heilmann, http://icant.co.uk
YQL and YUI: Building Blocks for Quick Applications Video of Chris Heilmann talking about how to use YQL and YUI to simplify Web development. Chris Heilmann, YDN Theater
YQL Console Update - Paul Donnelly Paul Donnelly from the YQL team discusses changes to the YQL console including Query Aliasing. Paul Donnelly, YDN