Response Data

In this Chapter:

Supported Response Formats

The YQL Web Service can return data in either XML, JSON, or JSONP format. The default format is XML. To specify JSON, include the format=json parameter in the URL of the YQL Web service, for example:

http://query.yahooapis.com/v1/public/yql?q=select * from geo.places wheretext=’sunnyvale’&format=json

To specify JSONP, include both the format and callback query parameters. The callback parameter indicates the name of the JavaScript callback function. Here’s an example:

http://query.yahooapis.com/v1/public/yql?q=select * from geo.places wheretext=’sunnyvale’&format=json&callback=cbfunc

The format of the response data is not dependent on the format of the original datasource. For example, if a YQL table is backed by a datasource in XML, the YQL Web Service can return data in JSON. For more information, see XML-to-JSON Transformation.

JSONP-X: JSON envelope with XML content

Aside from offering JSON as a response format with callbacks, you can also specify XML as the response format. If in your query you specify a callback (callback=cbfunction) and also request the format be in XML (format=xml), then YQL returns a string representation of the XML within an array. Compare the following Yahoo Local search for Indian restaurants in Sunnyvale, California using JSONP and JSONP-X callbacks, respectively:

JSONP Callback

cbfunction({"query":{"count":"10","created":"2009-07-10T09:13:28Z","lang":"en-US","updated":"2009-07-10T09:13:28Z",
  "uri":"http://query.yahooapis.com/v1/public/yql?q=select+Title+from+local.search+where+zip%3D%2794085%27+and+query%3D%27indian+restaurants%27",
  "diagnostics":{"publiclyCallable":"true",
  "url":{"execution-time":"332",
  "content":"http://local.yahooapis.com/LocalSearchService/V3/localSearch?zip=94085&query=indian%20restaurants&start=1&results=10"},
  "user-time":"335","service-time":"332","build-version":"2213"},"results":{"Result":[
  {"Title":"Grand Indian Buffet"},
  {"Title":"Turmeric Restaurant"},
  {"Title":"Taj India"},
  {"Title":"Shalimar"},
  {"Title":"Komala Vilas"},
  {"Title":"Brindavan Fine Indian Cuisine"},
  {"Title":"Panchavati Indian Veggie Foods"},
  {"Title":"Sneha Restaurant"},
  {"Title":"Bhavika's Food to Go"},
  {"Title":"ATHIDHI INDIAN CUISINE"}]}}});

JSONP-X Callback

cbfunction({"query":{"count":"10","created":"2009-07-10T09:10:29Z","lang":"en-US","updated":"2009-07-10T09:10:29Z",
  "uri":"http://query.yahooapis.com/v1/public/yql?q=select+Title+from+local.search+where+zip%3D%2794085%27+and+query%3D%27indian+restaurants%27",
  "diagnostics":{"publiclyCallable":"true",
  "url":{"execution-time":"558",
  "content":"http://local.yahooapis.com/LocalSearchService/V3/localSearch?zip=94085&query=indian%20restaurants&start=1&results=10"},
  "user-time":"561","service-time":"558","build-version":"2213"}},"results":[
  "<Result xmlns=\"urn:yahoo:lcl\"><Title>Grand Indian Buffet<\/Title><\/Result>",
  "<Result xmlns=\"urn:yahoo:lcl\"><Title>Turmeric Restaurant<\/Title><\/Result>",
  "<Result xmlns=\"urn:yahoo:lcl\"><Title>Taj India<\/Title><\/Result>",
  "<Result xmlns=\"urn:yahoo:lcl\"><Title>Shalimar<\/Title><\/Result>",
  "<Result xmlns=\"urn:yahoo:lcl\"><Title>Komala Vilas<\/Title><\/Result>",
  "<Result xmlns=\"urn:yahoo:lcl\"><Title>Brindavan Fine Indian Cuisine<\/Title><\/Result>",
  "<Result xmlns=\"urn:yahoo:lcl\"><Title>Panchavati Indian Veggie Foods<\/Title><\/Result>",
  "<Result xmlns=\"urn:yahoo:lcl\"><Title>Sneha Restaurant<\/Title><\/Result>",
  "<Result xmlns=\"urn:yahoo:lcl\"><Title>Bhavika's Food to Go<\/Title><\/Result>",
  "<Result xmlns=\"urn:yahoo:lcl\"><Title>ATHIDHI INDIAN CUISINE<\/Title><\/Result>"]});

Structure of Response

Every response from YQL includes a query element, which contains the diagnostics and results elements. (For details on the diagnostics element, see Information About the YQL Call.) The repeating elements within result are “rows” from a YQL table. For example, select * from social.connections returns multiple connection elements within the result element.

The following listing shows the basic structure of the XML data in the response of a call to the YQL Web Service.

<query ...  (attributes such as count)>
    <diagnostics>
        ... (sub-elements such as publiclyCallable)
    <results>
        ... (data returned by the call to YQL)
    </results>
</query>

The next listing shows the basic structure of YQL response in JSON format:

{
  "query": {
     "count": ...
     ...
     "diagnostics": {
        "publiclyCallable": ...,
        ...
     },
     "results": {
       // data returned by call to YQL
       ...
}

Information About the YQL Call

To get information about the execution of the YQL call, check the attributes of the query element and the sub-elements of the diagnostics element.

The following table lists the attributes of the query element in an XML response. In a JSON response, these attributes are mapped to the name-value pairs contained in the query object.

Attribute of query Element Description
count The number of items (rows) in returned by the YQL statement. In an XML response, count is the number of sub-elements in the results element.
created The date and time the response was created.
lang The locale for the response.
updated The date and time this response was last updated.

The diagnostics element contains information about the calls the YQL Web service made to back-end datasources. The following table lists the XML sub-elements of the diagnostics element. In a JSON response, these sub-elements are mapped to name-value pairs contained in the diagnostics object.

Sub-element of diagnostics Element Description
publiclyCallable True if the table is public data, false for private data. Authorization is required for private data.
url The URL of the Web service called by YQL to get the data. The value of the execution-time attribute is elapsed time, in milliseconds required to call the URL.
user-time The time YQL takes to perform the request and return the result, including time taken to execute Javascript. This is the time that you wait for each YQL statement response.
service-time The time YQL would take if each request were sequentially performed. YQL performs most request-related actions in parallel, so this measurement is informational only.

XML-to-JSON Transformation

If the YQL results are in JSON format, and the table is backed by an XML data source, then YQL transforms the data from XML to JSON. This transformation is “lossy,” that is, you cannot transform the JSON back to XML. YQL transforms XML data to JSON according to the following rules:

  • Attributes are mapped to name:value pairs.
  • Element CDATA or text sections are mapped to “content”:value pairs if the element contains attributes or sub-elements. Otherwise they are mapped to the element name’s value directly.
  • Namespace prefixes are removed from names.
  • If the attribute, element, or namespace-less element would result in the same key name in the JSON structure, an array is created instead.

For example, consider the following XML:

<doc yahoo:count=10>
   <ns:a>avalue</ns:a>
   <b><subb>bvalue</subb></b>
   <c count=20 yahoo:count=30>
      <count>40</count>
      <count><subcount>10</subcount></count>
   </c>
   <d att="cat">dog</d>
</doc>

This XML is transformed to the following JSON structure:

{ doc: {
  count:10,
     a:"avalue",
     b: { subb: "bvalue"},
     c: { count: [ 20,30,40,{subcount:10} ] },
     d: { att:"cat", content:"dog" }
}}

JSON-to-JSON Transformation

YQL transforms all JSON data sources into XML before returning results. To return JSON results that were obtained from a JSON data source, YQL must first transform the original JSON data to XML and then transform the XML back into a JSON result. During the tranformation from XML to JSON, the original JSON may be altered or become “lossy”. In other words, the original JSON may not be the same as the returned JSON.

The original JSON may be altered in the following ways:

  • JSON numbers are returned as strings.
  • JSON arrays containing a single element are returned as a JSON object.

To prevent this “lossy” transformation, you append the query string parameter jsonCompat=new to the YQL Web Service URL that you are using. For those creating tables, you use the jsonCompat(“new”) when making REST calls to other Web services. To illustrate how the jsonCompat parameter is used, we’ll look at the below examples that use the community table that queries the Google Books API, which only returns JSON.

  • Lossy JSON

    The following REST URI uses the public YQL Web Service URL and the Google Books table. The jsonCompat parameter is not added to the URI, so the original JSON returned from the Google Books API will be altered in the YQL response.

    https://query.yahooapis.com/v1/public/yql?q=SELECT * FROM google.books WHERE q="barack obama" AND maxResults=1&format=json&env=store://datatables.org/alltableswithkeys

    In the returned “lossy” JSON results below, notice that items is an object and that totalItems property has the the string value “958”.

    ...
      "results":{
        "json":{
          "kind":"books#volumes",
            "totalItems":"958",
            "items":{
              ...
            }
          }
        }
      }
    ...
    
  • Lossless JSON

    Appending the jsonCompat=new query parameter to the REST URI as seen below, YQL now returns the same JSON data as the Gowalla API.

    https://query.yahooapis.com/v1/public/yql?q=SELECT * FROM google.books WHERE q="barack obama" AND maxResults=1&format=json&env=store://datatables.org/alltableswithkeys&jsonCompat=new

    In the returned “lossless” JSON results below, the items property is now an array containing objects as elements, and totalItems is a number.

    ...
      "results":{
        "json":{
          "kind":"books#volumes",
          "totalItems":958,
          "items":[
            {
               ...
            }
          ]
        }
      }
    ...
    

Optimizing the Response

In the following sections, we’ll first look at the default response for a simple YQL statement and then examine how a modification to the same YQL statement can cause YQL to return a bloated response. Finally, we’ll show you how to configure YQL to return an optimized response.

Default Response

The simple YQL statement querying weather data below uses the filter location=90210 and requests all of the fields.

select * from weather.forecast where location=90210

This query returns a single response item, which contains two yweather:forecast sub-elements (one containing the forecast information for the current, and the other, the forecast information for the next day). In the default response shown below, notice that YQL places the yweather:forecast sub-elements in one item element:

<?xml version="1.0" encoding="UTF-8"?>
<query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng"    yahoo:count="1" yahoo:created="2012-07-02T17:34:20Z" yahoo:lang="en-US">
  <results>
    <channel>
      <title>Yahoo Weather - Beverly Hills, CA</title>
      <link>http://us.rd.yahoo.com/dailynews/rss/weather/Beverly_Hills__CA/*http://weather.yahoo.com/forecast/USCA0090_f.html</link>
      <description>Yahoo Weather for Beverly Hills, CA</description>
      <language>en-us</language>
      <lastBuildDate>Mon, 02 Jul 2012 9:49 am PDT</lastBuildDate>
      <ttl>60</ttl>
        ...
      <item>
        <title>Conditions for Beverly Hills, CA at 9:49 am PDT</title>
        <geo:lat xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#">34.08</geo:lat>
        <geo:long xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#">-118.4</geo:long>
        <link>http://us.rd.yahoo.com/dailynews/rss/weather/Beverly_Hills__CA/*http://weather.yahoo.com/forecast/USCA0090_f.html</link>
        <pubDate>Mon, 02 Jul 2012 9:49 am PDT</pubDate>              <yweather:condition xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="26" date="Mon, 02 Jul 2012 9:49 am PDT" temp="67" text="Cloudy"/>
        <yweather:forecast xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="32" date="2 Jul 2012" day="Mon" high="74" low="61" text="Sunny"/>
        <yweather:forecast   xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="30" date="3 Jul 2012" day="Tue" high="74" low="60" text="Partly Cloudy"/>
        <guid isPermaLink="false">USCA0090_2012_07_03_7_00_PDT</guid>
      </item>
    </channel>
  </results>
</query>

Bloated Response

The response will bloat though if a SELECT statement specifies one or more fields for projection, and those fields match repeated sub-elements with the same parent in the response. For the sake of clarification, let’s look again at our SELECT statement that queries weather data, its related response, and then make one modification to see how the response changes.

Here was our original SELECT statement querying weather data:

select * from weather.forecast where location=90210

Ignoring the actual returned data, let’s just look at the basic structure of the returned response shown below. Notice that the item element has two instances of the sub-element yweather:forecast.

<results>
  <channel>
    <title ... />
    <link ... />
    <description>
      ...
    </description>
    ...
    <item>
      <yweather:forecast ... />
      <yweather:forecast ... />
    </item>
  </channel>
</results>

Suppose we take the sub-element yweather:forecast and use it as a projection field so that our SELECT statement now looks like the following:

select item.yweather:forecast from weather.forecast where location=90210

The returned response for our modified query will look like this:

<results>
  <channel>
    <item>
      <yweather:forecast ... />
    </item>
  </channel>
  <channel>
    <item>
      <yweather:forecast ... />
    </item>
  </channel>
</results>

Using the sub-element yweather:forecast as a projection field caused the response to have multiple channel and item elements. This bloating would only increase if the query has title, link, and description set as projection fields because now the response will contain duplicate title, link, and description elements inside the repeated XML nodes.

For example, when we modify our query to request the projection fields title, link, and description with the following:

select title, link, description, item.yweather:forecast from weather.forecast where location=90210

You can see that our response becomes even more bloated:

<results>
  <channel>
    <title ... />
    <link ... />
    <description>
       ...
    </description>
    <item>
      <yweather:forecast ... />
    </item>
  </channel>
  <channel>
    <title ... />
    <link ... />
    <description>
       ...
    </description>
    <item>
       <yweather:forecast ... />
    </item>
  </channel>
</results>

In the following sections, we will look at how users can specify the crossProduct=optimized request parameter to configure YQL to optimize responses by removing redundant data.

Configuring YQL to Optimize the Response

In the Bloated Response section, we saw how the response for a one-day weather forecast contained redundant data when the SELECT statement specified nested fields as projection fields. Imagine how large the response for our weather query would be if it contained the forecast for the next 7 days.

Fortunately, YQL lets you append the request parameter crossProduct=optimized to the the YQL Web Service URLs to suppress the duplicate information from the response. To see how the request parameter crossProduct=optimized optimizes the response, click the link below to run the statement in the YQL console:

select title, link, description, item.yweather:forecast from weather.forecast where location=90210

Notice that the returned response contains a single item with two yweather:forecast sub-elements, with the title, link, and description elements (which is common across these sub-elements) appearing only once. This is because the YQL Web service URL had the appended query parameter crossProduct=optimized, which, from the YQL Console, you can see in the browser’s address bar or in the THE REST QUERY text field.

<?xml version="1.0" encoding="UTF-8"?>
<query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng"    yahoo:count="1" yahoo:created="2012-07-02T17:46:48Z" yahoo:lang="en-US">
  <results>
    <channel>
      <title>Yahoo Weather - Beverly Hills, CA</title>
      <link>http://us.rd.yahoo.com/dailynews/rss/weather/Beverly_Hills__CA/*http://weather.yahoo.com/forecast/USCA0090_f.html</link>
      <description>Yahoo Weather for Beverly Hills, CA</description>
      <item>
        <yweather:forecast xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="32" date="2 Jul 2012" day="Mon" high="74" low="61" text="Sunny"/>
        <yweather:forecast xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="30" date="3 Jul 2012" day="Tue" high="74" low="60" text="Partly Cloudy"/>
      </item>
    </channel>
  </results>
</query>

Note, only projection fields that are not also specified as local filters (which is true for all the projection fields in the above query) will respond to this type of optimization.

Errors and HTTP Response Codes

The YQL Web Service returns the below HTTP response codes. Open Data Table developers can also manipulate the HTTP response code that is returned to clients with the JavaScript method y.exit.

Error Description
200 OK The YQL statement executed successfully. If the YQL statement is syntactically correct and if authorization succeeds, it returns 200 OK even if the calls to back-end data services return 400 or 500 errors. Information about these back-end errors is in the diagnostics element of the response from YQL.
400 Bad Request Malformed syntax or bad query. This error occurs if the WHERE clause does not include a required input key. The XML error element includes a text description of the error. In the YQL Console, the error description appears in a highlighted bar.
401 Authorization Required The user running the application calling YQL is not authorized to access private data. This error also occurs if the user attempts to access his or her own private data without logging in to Yahoo.
999 - Unable to process this request at this time This error normally occurs when too many requests are sent to YQL and the rate limit has been reached.