Executing JavaScript in Open Data Tables

In this Chapter:

Introduction

Features and Benefits

The ability to execute JavaScript extends the functionality of Open Data Tables in many ways, including the following:

  • Flexibility beyond the normal templating within Open Data Tables: Executing JavaScript allows you to use conditional logic and to format data in a granular manner.
  • Better data shaping and parsing: Using JavaScript, you can take requests and responses and format or shape them in way that is suitable to be returned.
  • Better support for calling external Web services: Some Web services use their own security and authentication mechanisms. Some also require authentication headers to be set in the Web service request. The execute element allows you to do both.
  • Better support for adding, modifying, and deleting data using external Web services: For Web services that support write access, YQL allows you to insert, update, and delete using server-side JavaScript within the insert, update, and delete elements, which are nested within the binding element.
  • Ability to make asynchronous calls: YQL provides JavaScript methods for making REST calls that take a callback function parameter to handle the returned response. Using these methods with callback functions, you can initiate multiple calls that won’t be blocked by a slow response or a timed-out request.

The ability to execute JavaScript is implemented through the execute sub-element within an Open Data Table definition.

Within the execute sub-element, you can embed JavaScript and E4X (the shortened term for ECMAScript for XML), which adds native XML support to JavaScript. Support for E4X was first introduced in JavaScript 1.6.

When a YQL statement calls an Open Table Definition that contains the execute sub-element, YQL no longer performs the request to the templated URI in the endpoint. Instead YQL provides a runtime environment in which the JavaScript is executed server-side. Your JavaScript in turn must then return data as the output to the original YQL statement.

Ensuring the Security of Private Information

As mentioned earlier, a important feature of Open Data Tables is the ability to accommodate third-party security and authentication systems. As such, it is critical for developers to ensure an HTTPS connection is required in any case where “secret” or “private” information is being provided.

If your table requires input that is deemed “private”, such as any passwords, authentication keys, or other “secrets”, you MUST ensure the https attribute within the table element is set to true.

When YQL detects the https attribute is set to true, the table will no longer be usable for connections to the YQL console or to the Web service API. To test and use tables securely, you should now use the HTTPS endpoints:

Note

Connections made from the Open Data Table to the underlying Web services do not need to be made over HTTPS. The same holds true for the actual server hosting the Open Data Table definition.

For more information on the https attribute within Open Data Tables, refer to “tables element” section within Using Open Data Tables.

JavaScript Objects, Methods, and Variables Reference

As you add JavaScript within your execute sub-element, you can take advantage of the following global objects:

Object Description
y Global object that provides access to additional language capabilities.
request The request object is a global reference to the rest object that contains the URL on which YQL would normally perform a GET request. You cannot reference the rest object directly, so you need to use request or y.rest to access the rest object.
response Response object returned as part of the “results” section of the YQL response.

Let’s discuss each of the global objects in detail.

y Global Object

The y global object contains methods that provide basic YQL functionality within JavaScript. It also allows you to include YQL Open Data Tables and JavaScript from remote sources.

Method/Property Description Returns
``cache`    
context Provides information about the context or environment that JavaScript code is running in. Yahoo engineers have the additional properties context.env to access the environment and context.host to access the physical hostname in which the statement is executed. The possible environments are prod, gamma, or dev. The dev environment is only available to table authors running their own YQL engine. JavaScript object
context.host The name of the host server that is executing the YQL statement. string
context.table The name of the Open Data Table where JavaScript within an execute element is running. string
createTenantContext(tenantName) Creates a tenant context that can be used to execute YQL statements. JavaScript object
crypto    
date Provides access to date-related functions. JavaScript object
decompress(base64_compressed_string) Decodes a base64 string and then decompresses that string with gunzip. This method allows you to decompress the POST request body and query parameters. string
deflate(string, level) Deflates a string at the specified compression level and then Base64-encodes the result. Base64-encoded and compressed string
diagnostics Returns diagnostic information related to the currently executed script. Diagnostic information
env(environment file) Includes an environment file for use with your JavaScript. Example: ``y.env(“http://datatables.org/alltables.env”); ``
``exit(), exit(status_code,msg) ``    
getTenantContextsAvailable() Gets the names of tenants whose contexts are available. JavaScript object
include(url) Includes JavaScript located at a remote URL. Returns an evaluation of that include.
inflate(base64_deflated_string) Decodes a Base64-encoded compressed string and then inflates it. string
log(message) Creates a log entry in diagnostics. Returns the log entry within the diagnostics output associated with the current select statement
query(statement,params, timeout, callback) Prepares and runs a YQL statement. Execute will replace all ``@name ``fields in the YQL statement with the values corresponding to the name in the supplied hash table. Creates a result instance, or returns an error.
rest(url, callback) Sends a GET request to a remote URL endpoint.
sync(flag) Defers the JavaScript execution until pending asynchronous requests have been completed.
tidy (html) Tidy and return provided HTML. HTML that is run through HTML Tidy.
use(url,namespace) Imports an external Open Data Table definition into the current script at runtime.
xpath(object,xpath) Applies XPath to an E4X object. E4X object

Conversion Methods

Method/Property Description Returns
jsonToXml(json_obj) Converts a JavaScript/JSON object into E4X/XML. E4X object
jsToString(json_object) Converts a JSON object to a string. string
xmlToJson(object) Converts an E4X/XML object into a JSON object. JavaScript object
xparseJson(json_str) Returns a JavaScript object when given a well-formed JSON string. JavaScript object

y.cache

The y.cache object has methods for getting, setting, removing, incrementing, and decrementing cache values. These methods all return a CacheOpResult object that can be used to find out the cache key specified for the operation, the cache operation being carried out, the result of the cache operation, and the type of cache being used (either memecache or localcache). The CacheOpResult object is essentially a handle on the async cache operation that is being carried out.

cache Object

Method/Property Description Returns
CacheOpResult The object provides access to methods the operation being performed, the type of caching being used, and the cache value returned. JavaScript object
decr(key, by, def, expiry [, timeout]) Decrements the value associated with a cache key by a specified amount. CacheOpResult object
get(key [, timeout]) Returns a CacheResult object that can be used to get the value associated with the cache key. CacheOpResult object
``incr(key, by, def, expiry [, timeout]) Increments the value associated with a cache key by a specified amount. CacheOpResult object
``put(key, value [,expiry]) Sets a value for the given cache key. CacheOpResult object
remove(key [, timeout]) Removes the given key from the cache. CacheOpResult object

CacheOpResult Object

Property Description Returns
op The name of the operation, such as "get". string
result For the get method, the result object contains the value associated with the cache key. For other methods, a boolean value is returned indicating whether the method was successful. boolean or string
type The type of cache being used. The two possible values are "memcached" or "localcache". The type of cache being used. The two possible values are "memcached" or "localcache".

Methods

decr(key, by, def, expiry [, timeout])

Decrements the numeric value of the given key by the amount specified by the parameter by. If the key is not found, the key will be inserted into the cache with the value of the parameter def. If the value of key exists in the cache, but there was a problem in decrementing its value, then the key will be initialized with new value of def in the cache.

Parameters:

  • key <String> The cache key.
  • by <Number> The number to subtract from the initial value.
  • def <Number> The initial value for the key if the key does not exist in the cache yet.
  • expiry <Number> The number of seconds that the key will remain in the cache. If no value is given, the value remains in the cache for 2,500 seconds.
  • timeout <Number> (Optional) The number of milliseconds that the operation must complete within before timing out.

Returns: CacheOpResult object

Examples:

// Insert a value in cache that will be incremented later. // Note that the numeric value is inserted
// as a string so that y.cache.incr and y.cache.decr can
// properly operate on the value.

// Blocking call
var asyncop_a = y.cache.put("test2", "10", 100).result;

// Increment the previously inserted value.
var asyncop_b = y.cache.incr("test2", 2, 0, 100);
// get the result of the async incr operation

// Blocking call
y.log("incr operation: " + a.result);
get(key [, timeout])

Returns a CacheOpResult object containing the value associated with the given key from the cache.

Parameters:

  • key <String> The cache key.
  • timeout <Number> (Optional) The number of milliseconds that the operation must complete within before timing out.

Returns: CacheOpResult object

Examples:

// Retrieve value from cache asynchronously
// (returns CacheOpResult object)
var asyncop = y.cache.get("key1");

// Non-blocking operation
if (asyncop.result) {
  // Blocks
  // Value retrieved successfully
} else {
  // Value was not found in cache
}

// Retrieve multiple values asynchronously
var getops = [];
for (var i = 0; i < 11; ++i) {
  getops.push(y.cache.get("test" + i, 100));
}

// Get values.
data += "Get Values:\n";
for (var i = 0; i < 11; ++i) {
  data += "Result: " + getops[i].result + ", Type: " + getops[i].type + ", Op: " + getops[i].op + ", Key: " + getops[i].key + "\n";
}
incr(key, by, def, expiry [, timeout])

Increases the numeric value of the key by the amount specified by the parameter by. If the key is not found, the key will be inserted into the cache with the value of the parameter def. If the value of key exists in the cache, but there was a problem in decrementing its value, then the key will be initialized with new value of def in the cache.

Parameters:

  • key <String> The cache key.
  • by <Number> The number to add to the initial value.
  • def <Number> The initial value associated with the key if it does not exist in the cache already.
  • expiry <Number> The number of seconds that the key will remain in the cache. If no value is given, the value remains in the cache for 2,500 seconds.
  • timeout <Number> The number of the milliseconds for the timeout.

Returns:

CacheOpResult object

Examples:

// Insert a value in cache that will be incremented later. // Note that the numeric value is inserted
// as a string so that y.cache.incr and
// y.cache.decr can properly operate on the value.
var asyncop_a = y.cache.put("test2", "10", 100).result;

// Blocking call
// Increment the previously inserted value.
var asyncop_b = y.cache.incr("test2", 2, 0, 100);

// Get the result of the async incr operation
y.log("incr operation: " + a.result); // blocking call
put(key, value [, expiry])

Sets the value of the given key into the cache.

Parameters:

  • key <String> The cache key.
  • value <String> The value to set for the cache key.
  • expiry <Number> (Optional) Number of seconds that the value remains in cache. The default value is 2500.

Returns:

CacheOpResult object

Examples:

// Put a value in cache asynchronously (returns CacheOpResult object)
var asyncop = y.cache.put("key1", "value1");

// Non-blocking call.
// Confirm that value associated with 'key1'
// was inserted successfully in cache
if (asyncop.result) {
  // asyncop.result is blocking
  // value inserted sucessfully
} else {
  // Could not insert value for some reason.
}

// If blocking call is ok, then the above
// operation can be carried out in single step.   y.cache.put("key1", "value1").result;

// Blocking call
// async operation can also be used to insert
// multiple values quickly without any waiting time.
var putops = [];
for (var i = 0; i < 10; ++i) {
  putops.push(y.cache.put("test" + i, i*10, 100));
}

// Examine the state and wait
var data = "Confirm puts:\n";
// Check if put operations succeeded
for (var i = 0; i < 10; ++i) {
  data += "Result: " + putops[i].result + ", Type: " + putops[i].type + ", Op: " + putops[i].op + ", Key: " + putops[i].key + "\n";
}
remove(key [, timeout])

Removes a key and its value from the cache.

Parameters:

  • key <String> The cache key.
  • timeout <Number> (Optional) The number of milliseconds that the operation must complete within before timing out.

Returns:

CacheOpResult object

Examples:

// Remove value from cache asynchronously
// (returns CacheOpResult object)
var asyncop = y.cache.remove("key1");
// Check if value was remved successfully
if (asyncop.result == true) {
  // Blocking call
  // value removed successfully
} else {
  // Could not remove value
}

y.cache (Deprecated)

As this version of y.cache has been deprecated, please see the new implementation of y.cache.

The y.cache object has methods for getting, setting, removing, incrementing, and decrementing cache values.

Deprecation Reasons

We implemented a new cache API because the old caching API has the following limitations:

  • y.cache functionality is not exposed to external tables.
  • y.cache.get should be async, so that it is possible to execute several y.cache.get calls in parallel (same with y.cache.put).
  • y.cache.put currently wraps all values in CacheItem object and then stores the CacheItem object in actual cache. As a result, y.cache.incr and y.cache.decr operations are not compatible with y.cache.put and y.cache.get operations.
  • No way to check if async operations y.cache.put and y.cache.remove actually succeeded without making a subsequent sync call to y.cache.get to confirm.
  • y.cache.get returns “null” when a key is not present in the cache and the actual value associated with the key is null.

Backwards Compatibility

YQL tenants can be migrated from old y.cache API to the new y.cache api by setting the cacheAsyncByDefault tenant configuration property to true. Alternatively, the new y.cache API may be enabled on a per-request basis, by appending the query parameter cacheAsyncByDefault=true to the YQL request. Functionally, the new y.cache API is a superset of the old y.cache API in that all the functionality of old API is present in the new API but not vice versa. With respect to the usage syntax, the new y.cache API is very different from the old y.cache API.

These two issues (functionality and syntax) make the old and new y.cache API incompatible with each other. Code migration (very simple in most cases) is required to move the old y.cache javascript code to new y.cache API. We plan on rolling out the new y.cache api gradually across all tenants.

Methods

decr(key, step, init, timeOut)

Sets a timeout and decreases the numeric value mapped to a key by a specified integer.

Parameters:

  • key <String> The cache key.
  • step <Number> The number to subtract from the initial value.
  • init <Number> The initial value for the cache key.
  • timeOut <Number> Number of seconds before timeout.

Returns: Number

Examples:

var cache_decr = y.cache.decr("user", 2, 12.5, 2);
get(key)

Returns the value associated with a key.

Parameters:

  • key <String> The cache key.

Returns: Object

Examples:

var cache_user = y.cache.get("user");
y.log("User: " + cached_user);
get(key,timeOut)

Sets a timeout and returns the value associated with a key.

Parameters:

  • key <String> The cache key.
  • timeout <Number> Number of seconds before timeout.

Returns: Object

Examples:

var cache_user = y.cache.get("user",2);
y.log("User: " + cached_user);
incr(key, step, init, timeOut)

Sets a timeout and increases the numeric value mapped to a key by a specified integer.

Parameters:

  • key <String> The cache key.
  • step <Number> The number to add to the initial value.
  • init <Number> The initial value associated with the cache key.
  • timeout <Number> The integer of the milliseconds for the timeout.

Returns: Object

Examples:

var cache_incr = y.cache.incr("user", 5, 10.5, 2);
put(key, value, timeOut)

Sets a timeout and the value associated with a key.

Parameters:

  • key <String> The cache key.
  • value <String> The value associated with the cache key.
  • timeout <Number> Number of seconds before timeout.

Returns: None

Examples:

y.cache.put("user","john_doe",2);
remove(key)

Removes a key and its value from the cache.

Parameters:

  • key <String> The cache key.

Returns: None

Examples:

var cache_user = y.cache.remove("user");

y.crypto functions

YQL provides several cryptographic functions for use within JavaScript. These functions reduce the need for external libraries and make YQL easier to use.

Examples:

var md5string = y.crypto.encodeMd5("encode this string to md5");
Function Description Returns
encodeHmacSHA256(String secret, String plaintext) Encrypts a string using HMAC-SHA256 encryption. Returns an encrypted string.
``encodeHmacSHA1(String secret, String plaintext) `` Encrypts a string using HMAC-SHA1 encryption. Returns an encrypted string.
encodeMd5(String plaintext) Provides the MD5 hash of a string. Returns an MD5 hash of a BASE64-encoded string.
encodeMd5Hex(String plaintext) Provides the MD5 hash of a hex-encoded string. Returns an MD5 hash
encodeSha(String plaintext) Provides the SHA-1 hash of a string. Returns an SHA-1 hash.
encodeBase64(String plaintext) Performs Base64 encoding of a string. Returns an Base64 encoded string.
decodeBase64(String plaintext) Performs Base64 decoding of a string. Returns an Base64 decoded string.
``uuid() `` Provides a cryptographically secure version 4 Universal Unique Identifier (UUID). Returns a UUID.

y.date

Using y.date gives you access to date-related functions.

Methods

y.date.getOffsetFromEpochInMillis(time_format)

Parses the given date string in a format specified by RFC 3339 and returns the number of milliseconds since the Unix epoch (0:00:00 UTC on January 1, 1970) as a string.

Parameters:

  • time_format <String> A string in the form of the time/date formats or regex patterns below. The regex patterns are case insensitive.
    • RFC 3339 Timestamp
    • ^([+-])?\s*(\d+)\s*(year|month|week|day|hour|minute)s?$
    • ^(\d+)\s*(year|month|week|day|hour|minute)s?\s+(from\s+now|after|after\s+today|later|old|ago)$
    • ^now\s+([+-])\s*(\d+)\\s+(year|month|week|day|hour|minute)s?$
    • ^last\s+(year|month|week)$
    • now
    • yesterday
    • tomorrow

Returns: String

Examples:

In the example table below, getOffsetFromEpochInMills is used in conjunction with the JavaScript Date object to log several different times. The table then returns the execution time of the JavaScript in the response.

...
  <execute><![CDATA[
    var start_time = parseInt(y.date.getOffsetFromEpochInMillis("now"));
    var start_date = new Date(start_time);
    var yesterday_time =  parseInt(y.date.getOffsetFromEpochInMillis("yesterday"));
    var yesterday_date =  new Date(yesterday_time);
    var month_from_now =  new Date(parseInt(y.date.getOffsetFromEpochInMillis("now + 1 month")));
    var ali_liston_fight = new Date(parseInt(y.date.getOffsetFromEpochInMillis("1964-02-25")));
    y.log("Start Time: " + start_date.toDateString());
    y.log("Yesterday: " + yesterday_date.toDateString());
    y.log("Last Month: " + month_from_now.toDateString());
    y.log("Ali beats Liston: " + ali_liston_fight.toDateString());
    response.object =  "Execution Time: " + String((parseInt(y.date.getOffsetFromEpochInMillis("now")) - parseInt(start_time))/1000) + " seconds.";
  ]]>
  </execute>
...

In this example XML response returned by the above table, the diagnostic information includes the logged times, and the result element contains the execution time.

<query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng"    yahoo:count="1" yahoo:created="2011-07-12T02:20:59Z" yahoo:lang="en-US">
  <diagnostics>
    <publiclyCallable>true</publiclyCallable>
    <url execution-time="76">
      <![CDATA[http://zhouyaoji.com/yql/date.xml]]>
    </url>
    <log>Start Time: Tue Jul 12 2011</log>
    <log>Yesterday: Mon Jul 11 2011</log>
    <log>Last Month: Fri Aug 12 2011</log>
    <log>Ali beats Liston: Tue Feb 25 1964</log>
    <javascript execution-time="6" instructions-used="12134" table-name="sc"/>
    <user-time>89</user-time>
    <service-time>76</service-time>
    <build-version>19521</build-version>
  </diagnostics>
  <results>
    <result>Execution Time: 0.004 seconds.</result>
  </results>
</query>

y.decompress(base64_compressed_string)

Allows you to decompress query parameters and the HTTP POST request body. The decompress method decodes a base64 string and then uses gunzip to decompress the string.

Parameters:

  • base64_compressed_string <String> A string that has been compressed with gzip and then base64 encoded.

Returns: String

Examples:

YQL Statement:

use 'http://example.com/decompress.xml' as decompress; select * from decompress where foo="H4sIAOQTlU0AA8tIzcnJBwCGphA2BQAAAA=="

Example YQL Open Data Table: http://example.com/decompress.xml

The table decompress.xml below uses y.decompress to decompress the value assigned to foo in the YQL statement above and then returns it as the string.

<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd" securityLevel="any">
  <bindings>
    <select itemPath="" produces="XML">
      <inputs>
        <key id="foo" type="xs:string" paramType="variable"required="true"/>
      </inputs>
      <execute><![CDATA[
        try {
          y.log(foo);
          var bar = y.decompress(foo);
        } catch (err) {
          y.log(err.message);
        }
        response.object = bar;
      ]]></execute>
    </select>
  </bindings>
</table>

Returned XML Response:

<?xml version="1.0" encoding="UTF-8"?>
<query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng" yahoo:count="1"yahoo:created="2011-04-01T00:06:38Z" yahoo:lang="en-US">
  <diagnostics>
    <publiclyCallable>true</publiclyCallable>
    <url execution-time="2"><![CDATA[http://example.com/decompress.xml]]></url>
    <log>H4sIAOQTlU0AA8tIzcnJBwCGphA2BQAAAA==</log>              <javascript execution-time="11" instructions-used="0"table-name="d"/>
    <user-time>42</user-time>
    <service-time>2</service-time>
    <build-version>12012</build-version>
  </diagnostics>
  <results>
    <result>hello</result>
  </results>
</query>

y.deflate(string, level)

Deflates the given string into a zlib format at the specified compression level and then Base64-encodes the result.

Parameters:

  • string <String> - The input string to compress.
  • level <Number> - The compression level. See Class Deflater for the list of supported compression levels.

Returns: String

Examples:

In the example below, data is compressed with y.deflate and then posted to a Web service with y.rest.

...
  <execute><![CDATA[
    var data = "{ customer: { name: 'John Lucas' }}";
    var compressed_data = y.deflate(data, 1);
    var ret = y.rest("http://example-webservice.com").post(compressed_data).response;
    response.object = ret;
  ]]></execute>
...

y.exit(), y.exit(status_code,msg)

Causes the script to exit. If no parameters are given to y.exit, the YQL engine returns the default HTTP response status code and message to the client. The table author can change the HTTP response code sent to the client by passing a status code and a message as parameters to y.exit.

Parameters:

  • status_code <Number> - The HTTP response status code that gets returned to the client.
  • msg <String> - The message that gets returned to the client.

Returns: None

Example:

...
  <execute><![CDATA[
    var myRequest = y.rest('http://example.com');
    var data = myRequest.get().response;
    if(!data){
      y.exit(204, "Sorry, no content.");
    } else {
      response.object = data;
    }
  ]]></execute>
...

y.inflate(base64_deflated_string)

Decodes the given Base64-encoded compressed string and and then inflates it.

Parameters:

  • base64_deflated_string <String> - The input string to decode and inflate.

Returns: String

Examples:

In the example below, a Base64-coded and deflated string is passed as a query string parameter to the YQL Web Service URI. The table decodes and inflates the data, processes the inflated data, and then posts the data to the Web service.

...
  <insert itemPath="status" produces="JSON">
    <urls>
      <url>http://api.example.com/update/page/{id}</url>
    </urls>
    <inputs>
      <key id="id" type="xs:string" paramType="variable" required="true"/>
      <key id="data" type="xs:string" paramType="variable" required="true"/>
    </inputs>
    <execute><![CDATA[
      // Decode and inflate data
      var inflated_data = y.inflate(data);
      // Process inflated data
      var lines = inflated_data.split("\n");
      // Post inflated data
      response.object = y.rest(request.url).post(inflated_data);
      ]]>
    </execute>
  </insert>
...

y.jsonToXml(json_object)

Converts a JSON object into its equivalent XML (E4X) object.

Parameters

  • json_object <Object> A JSON object.

Returns: E4X object

// JSON object
var json_obj = { apis: [ "YQL", "Search", "Flickr"], libraries: { "js": [ "YUI" ] } };
var e4x = y.jsonToXml(json_obj);
// Displays the following:
//  <result>
//    <libraries>
//      <js>YUI</js>
//    </libraries>
//    <apis>YQL</apis>
//    <apis>Search</apis>
//    <apis>Flickr</apis>
//  </result>
y.log(e4x);

y.jsToString(json_object)

Converts a JSON object into its equivalent JSON string form.

Parameters:

  • json_object <Object> A JSON object.

Returns: String

// JSON object
var json_obj = { apis: [ "YQL", "Search", "Flickr"], libraries: { "js": [ "YUI" ] } };
var json_str = y.jsToString(json_obj);
//  Displays the following: // {"libraries":{"js":["YUI"]},
// "apis":["YQL","Search","Flickr"]}
y.log(json_str);
response.object =  <json_string>{json_str}</json_string>;

y.query(statement, params, timeout, callback)

Perhaps you want to use YQL queries while still using JavaScript within YQL. y.query allows you to perform additional YQL queries within the execute sub-element.

Parameters:

  • statement <String> A YQL statement.

  • params <Object> The object can contain key-values that can be referenced in the YQL statement for variable substitution. In the example below, the @query, @cat_id, and @type variables are passed to the YQL statement and replaced with the values associated with the keys of the params object.

    var q = y.query("select * from answers.search where query=@query and category_id=@cat_id and type=@type", {query:"cars",cat_id:"2115500137",type:"resolved"});
    var results = q.results;
    
  • timeout <Number> The number of milliseconds before the call times out.

  • callback <Function> The callback function to handle the returned response. See Making Asynchronous Calls with JavaScript Execute for more information about using the callback.

Returns: Object

The returned object contains the following properties:

Property Description Data Type
diagnostics The diagnostic information associated with the request. E4X object
format Allows you to request results to be returned as JSON or XML. The default format is XML. function
query The YQL statement that was passed as a parameter to y.query. string
results The results returned by the query. E4X object
timeout Specifies the request timeout in milliseconds. This is useful when you want to cancel requests that take longer than expected. None

Examples:

In the following code snippet that we looked earlier, y.query executes a statement that uses the answers.search table to get answers for questions regarding cars. The results are returned in the default format XML.

var q = y.query("select * from answers.search where query=@query andcategory_id=@cat_id and type=@type",{query:"cars",cat_id:"2115500137",type:"resolved"});
// Results returned in XML, the default
var results = q.results;

Using the format method, you can have the results returned in JSON:

var q = y.query("select * from answers.search where query=@query andcategory_id=@cat_id and type=@type",{query:"cars",cat_id:"2115500137",type:"resolved"}).format("json");
// Results returned as JSON
var results = q.results;

y.rest(url [, callback])

The y.rest method allows you to make GET requests to remote Web services. It also allows you to pass parameters and headers in your request and use a callback function to make asynchronous requests. See Making Asynchronous Calls with JavaScript Execute for more information about using the callback.

Parameters:

  • url <String> The URL endpoint to a query.
  • callback <Function> (Optional) The callback mechanism waits to receive returned results that it can then process. This callback function allows asynchronous calls because the function can wait for the returned results while y.rest is called again.

Returns: rest Object

Examples:

In the following code snippet, an HTTP GET request is made to example.com and the response is saved to data.

var myRequest = y.rest('http://example.com');
var data = myRequest.get().response;

The two properties url and timeout of the response help you determine if a call has timed out or associate the response with the original URL. The code snippet below show you the url and timeout properties and how you could potentially use them:

// The 'url' property allows you to connect
// the original request url to the response
var uri = y.rest("http://www.yahoo.com").get().url

// The 'timeout' property allows you to make another call
// or take other action if your original call times out
var timed_out = y.rest('http://www.yahoo.com').get().timeout

if(timed_out) {
  // Try one more time
   var resp = y.rest(uri).get().response;
}

Tip

The y.rest method supports “chaining”, which means that you can construct and run an entire REST request by creating a “chain” of methods. Here is a hypothetical example:

var myData = y.rest('http://blah.com')
          .path("one")
          .path("two").query("a","b")
          .header("X-TEST","value")
          .get().response;

When chained, the resulting request looks like this:

http://blah.com/one/two?a=b

Note

Because JSON does not have a “root” node in most cases, all JSON responses from a remote Web service will be contained within a special json root object under response.results.

y.sync(flag)

Causes the JavaScript execution to wait until pending y.rest and y.query asynchronous requests have been completed. If asynchronous y.rest and y.query requests refer to a JavaScript callback function, then y.sync should be called to ensure that JavaScript execution does not complete before all the asynchronous requests have completed and before callback functions have been invoked.

Returns true if there are no more asynchronous operations pending, otherwise returns false.

Parameters:

  • flag <Boolean> If true, y.sync will block until all asynchronous operations have completed. Otherwise, y.sync will block until any one of the pending asynchronous operations have either completed, timed out, or produced errors.

Returns: Boolean

Examples:

See Making Asynchronous Calls with JavaScript Execute for examples.

y.xmlToJson(e4x_obj)

Converts a E4X object into its equivalent JSON string form.

Parameters:

  • e4x_object <Object> An E4X object.

Returns: String

// E4X object
var e4x_obj = <restaurants><italian><name>Tony's</name><name>Mangano's</name></italian><chinese><name>Chef Hu's</name><name>Hunan Garden</name></chinese></restaurants>;
// Convert E4X object to JSON object
var json = y.xmlToJson(e4x_obj);
// Displays the following:
//{"restaurants":{"italian":{"name":["Tony's","Mangano's"],
// "chinese":{"name":["Chef Hu's","Hunan Garden"]}}}
y.log(json);

y.xparseJson(json_str)

Parses a well-formed JSON string and returns the resulting JavaScript object. If given a malformed JSON string, y.xparseJson will throw a runtime JSONException.

Parameters:

  • json_str <String> A well-formed JSON string.

Returns: Object

Examples:

The table below parses a JSON string and returns a string value from one of the properties of the created JavaScript object.

<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd"securityLevel="any">
  <bindings>
    <select itemPath="" produces="XML">
      <inputs></inputs>
      <execute><![CDATA[
        var mycontent ='{\
           "firstName": "John",\
           "lastName": "Smith",\
           "address": {\
             "streetAddress": "21 2nd Street",\
             "city": "New York",\
             "state": "NY",\
             "postalCode": "10021"\
           },\
           "phoneNumbers": [\
             {\
               "type": "home",\
               "number": "212 555-1234"\
             },\
             {\
               "type": "fax",\
               "number": "646 555-4567"\
             }\
           ]\
         }';
         y.log(mycontent);
         j= y.xparseJson(mycontent);
         y.log(j);
         response.object=j['lastName'];
       ]]></execute>
     </select>
  </bindings>
</table>

Example XML Response:

<query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng"    yahoo:count="1" yahoo:created="2013-02-12T23:27:22Z" yahoo:lang="en-US">
  <diagnostics>
    <publiclyCallable>true</publiclyCallable>
    <log>
      {
        {
          "firstName": "John",
          "lastName": "Smith",
          "address": {
          "streetAddress": "21 2nd Street",
          "city": "New York",
          "state": "NY",
          "postalCode": "10021"
        },
        "phoneNumbers": [
          {
            "type": "home",
            "number": "212 555-1234"
          },
          {
            "type": "fax",
            "number": "646 555-4567"
          }
        ]
      }
    </log>
    <log>{"firstName":"John","lastName":"Smith","address":{"city":"NewYork","postalCode":"10021","state":"NY","streetAddress":"21 2ndStreet"},"phoneNumbers":[{"type":"home","number":"212555-1234"},{"type":"fax","number":"646 555-4567"}]}</log>
     <javascript execution-time="1" instructions-used="6000"table-name="xx"/>
   </diagnostics>
   <results>
     <result>Smith</result>
   </results>
 </query>

request Global Object

The request global object is essentially a reference to a rest object. You must use the request object, y.rest, or y.query to access the rest object.

rest Object

The rest object has properties and methods for getting information and making REST calls.

Properties

Property Description Data Type
headers Gets the hashmap of headers object
url Provides a URL endpoint to query string
queryParams Gets the hashmap of query parameters E4X object containing query parameters.
matrixParams Gets the hashmap of matrix parameters result object containing matrix parameters.

Methods

The rest object contains methods that provide basic YQL functionality within JavaScript.

Method Description Returns
accept(content-type) Specifies the type of content to send in the response using the Accept HTTP header. rest Object
contentType(content-type) Specifies the content-type of the data being sent. rest Object
decompress(bool) Configures REST call to decompress the returned response with gunzip. rest Object
del() Performs an HTTP DELETE. rest Object
fallbackCharset(charset_list) Overrides the list of fallback character sets, which is set to “utf-8, iso-8859-1” by default, for decoding the returned response. rest Object
filterChars(filter_list) Removes illegal characters from the response data based on specified encoding standards. rest Object
forceCharset(charset_list) Decodes the response using the first successful character set listed in charset_list. rest Object
get() Performs a GET request to the URL endpoint. result Object
head() Performs an HTTP HEAD request on a URL. JavaScript object
header(name, value) Adds an HTTP header to the request. rest Object
jsonCompat(mode) Allows you to get “lossless” JSON when making a REST call to a Web service. rest Object
matrix(name,value) Adds a matrix parameter to the request. rest Object
path(path_segment) Appends a path segment to the URI. request Object
post(content) Performs an HTTP POST, using the value of the content, to the URL endpoint. result Object
put(content) Performs an HTTP PUT, using the value of the content, to the URL endpoint. result Object
query(key, value) Adds a single query parameter. request Object
query(hashmap) Adds all the query parameters based on key-name hashmap. request Object
readTimeout(timeoutMillis) Adjusts the socket read timeout settings to wait longer (or shorter) for a response. rest Object
timeout(milli_seconds) Specifies the request timeout in milliseconds. None
accept(content-type)

Specifies the type of content to send in the response using the Accept HTTP header. This tells YQL what kind of data format you want returned, as well as how to parse it.

Parameters:

  • content-type <String> The content type of the data being sent. For example: application/xml.

Returns: rest Object

Examples:

The following is an example of how you would specify JSON as the return format for data you send as XML:

var ret =
request.accept('application/json').contentType('application/xml').post(content).response;

Using the above example, YQL will convert XML to JSON prior to returning it in the response.

contentType(content-type)

Specifies the content-type of the data being sent. An example of a content-type is: application/json. This object is useful with INSERT and UPDATE statements.

Note

This method does not automatically convert one data format to another prior to sending it, so if you indicate one format in contentType but actually send another, your Web service may produce an error.

Parameters:

  • content-type <String> The content-type of the data being sent. An example of a content-type is: application/json.

Returns: rest Object

The following is an example of how you would specify XML as the data you are sending:

var content = <employee>
  <name>John</name>
  <id>21</id>
</employee>;

var ret =
  request.contentType('application/xml').post(content).response;
decompress(bool)

Allows you to configure a REST call to decompress the returned response with gunzip.

Parameters:

  • boolean <Boolean>

Returns: rest Object

Examples:

In this code example, the header method sets the HTTP header Accept-Encoding to gzip, and then the decompress method is used to decompress the returned response.

var resp = y.rest("www.example.com").header("Accept-Encoding","gzip").decompress(true).get();
response.object = resp;
del()

Performs an HTTP DELETE. This object is useful with DELETE statements.

Parameters: None

Returns: result Object

Examples:

In the below Open Data Table, the del() method is used to delete the Mixi voice status identified by the post ID. Using del() sends an HTTP DELETE request to the URL defined in the url element.

<delete itemPath="" produces="JSON">
  <urls>
    <url>http://api.mixi-platform.com/2/voice/statuses/{post_id}</url>
  </urls>
  <inputs>
    <key id="oauth_token" type="xs:string" paramType="query" required="true" />
    <key id="post_id" type="xs:string" paramType="path" required="true" />
  </inputs><execute><![CDATA[response.object = request.del().response;]]></execute>
</delete>
fallbackCharset(charset_list)

Overrides the list of fallback character sets, which is set to “utf-8, iso-8859-1” by default, for decoding the returned response. YQL attempts to decode the response using the character sets listed in charset_list when the response either does not specify the character set or specifies an incorrect character set that results in a failed decoding.

Parameters:

  • charset_list <String> A list of one or more fallback character sets. For example: “ISO-8859-1, utf-8”

Response: rest Object

Examples:

response.object = y.rest(url).fallbackCharset('shift_jis,iso-8859-8, UTF-8, iso-8859-1').get().response;
filterChars(filter_list)

The filterChars(filter_list) method removes illegal characters from the response data based on the encoding standards passed as arguments.

Parameters:

  • filter_list - A string that contains one to three of the encodings listed below. The encodings are comma delimited, such as "encoding, xml, json".
    • encoding - removes any character that does not conform to Web service character encoding.
    • xml - removes any character that does not conform to the XML standard where conforming XML characters are are represented by c in the following list:
      • c == 0x09
      • c == 0x0A
      • c >= 0x20 && c <= 0xD7FF
      • c >= 0xE000 && c <= 0xFFFD
      • c >= 0x10000 && c <= 0x10FFFF
    • javascript - removes any character that does not conform to the JavaScript standard where non-conforming JavaScript characters are represented by c in the following list:
      • c == 0x0000 || c == 0x00ad
      • c >= 0x0600 && c <= 0x0604
      • c == 0x070f || c == 0x17b4 || c == 0x17b5
      • c >= 0x200c && c <= 0x200f
      • c >= 0x2028 && c <= 0x202f
      • c >= 0x2060 && c <= 0x206f
      • c == 0xfeff
      • c >= 0xfff0 && c <= 0xffff

Response: rest Object

Examples:

// Filters out characters that do not conform to the
// Web service, XML and JSON standards.
response.object = y.rest(url).filterChars("encoding, xml, json").get().response;
forceCharset(charset_list)

Forces YQL to decode the response using the first successful character set listed in charset_list. Using this method overrides both the character set specified by the response and the fallback character sets.

Parameters:

  • charset_list <String> A list of one or more character sets that YQL will use to decode the response. For example: “ISO-8859-1, utf-8”

Returns: rest Object

Examples:

y.rest('www.uol.com.br').forceCharset('iso-8859-1').get().response;
get()

Performs a GET request to the URL endpoint. This object is useful with SELECT statements.

Parameters: None

Returns: result Object

Examples:

 r = y.rest("yahoo.com").contentType('application/json').get();
response.object = r;
header(name, value)

Adds an HTTP header to the request.

Parameters:

  • name <String> The name of the HTTP header.
  • value <String> The value associated with the HTTP header.

Returns: rest Object

Examples:

The header method in this code snippet is used to pass authorization credentials to get a response.

y.include("http://yqlblog.net/samples/base64.js");
var authheader = "Basic "+Base64.encode(username+":"+password);
response.object = request.header("Authorization",authheader).get().response;

In this code snippet, the header method sets the HTTP header Content-Encoding to gzip, so that the returned response will be compressed.

var resp = y.rest("http://www.apache.org").header("Accept-Encoding","gzip").get();
response.object = resp.response;
jsonCompat(mode)

Allows you to get “lossless” JSON when making a REST call to a Web service. For more information about “lossy” JSON, see JSON-to-JSON Transformation and Preventing Lossy JSON Results.

Parameters:

  • mode <String> Currently, the only allowed value is “new”. In the future, other modes of lossless JSON compatibility might exist that require passing a different string to jsonCompat.

Returns: rest Object

Examples:

The table below makes two GET calls to the Yahoo Upcoming API. The first call returns lossy JSON, and the second call returns lossless JSON. You can view the difference in the returned JSON in the diagnostic logs.

<?xml version="1.0" encoding="UTF-8"?><table xmlns="http://query.yahooapis.com/v1/schema/table.xsd" securityLevel="any">
 <bindings>
    <select produces="JSON">
      <execute><![CDATA[
        var api_key = "<get an API key from Y! Upcoming - http://upcoming.yahoo.com/services/api/keygen.php";
        var url = "http://upcoming.yahooapis.com/services/rest/?api_key=" + api_key + "&method=event.search&search_text=concert&metro_id=1&format=json";
        var a = y.rest(url).get().response;
        var b = y.rest(url).jsonCompat('new').get().response;
        y.log(y.jsToString(y.xmlToJson(a)));
        y.log(y.jsToString(y.xmlToJson(b)));
        response.object = b;
      ]]></execute>
    </select>
  </bindings>
</table>

Note that to obtain lossless JSON when querying the above table, the client must append the query string parameter jsonCompat=new to the YQL URI as shown here:

http://query.yahooapis.com/v1/public/yql?q=use 'http://example.com/jsonCompat.xml as json.compat; select * from json.compat&format=json&jsonCompat=new

matrix(name,value)

Adds a matrix parameter to the request.

Parameters:

  • name <String>
  • value <String>

Returns: rest Object

Examples:

The code snippet below posts the matrix parameters to flower_order_server.com.

y.rest("http://flower_order_service.com").matrix('flower','roses').matrix('color','red').post();

Matrix parameters are attached to the URI resource like query parameters, but are delimited by a semicolon and not an ampersand and the matrix parameters are not separated from the URI resource by a question mark. The yql.rest call creates the following URL: http://flower_order_service.com;flower=roses;color=red

path(path_segment)

Appends a path segment to the URI.

Parameters:

  • path_segment <String> The path segment to add to a URI.

Returns: request Object

Examples:

var myData = y.rest('http://blah.com')
     .path("one")
     .path("two") .query("a","b")
     .header("X-TEST","value")
     .get().response;
post(content)

Performs an HTTP POST, using the value of the content, to the URL endpoint. This object is useful with INSERT, UPDATE, and DELETE statements.

Parameters:

  • content <String|Object> The data that is posted.

Returns: result Object

Examples:

var content = { "employee": { "name": "John", "id": 21 }};
var ret = request.contentType('application/json').post(content).response;
put(content)

Performs an HTTP PUT, using the value of the content, to the URL endpoint. This object is useful with INSERT, UPDATE, and DELETE statements.

Parameters:

  • content <String | Object> The data that is used in a PUT request.

Returns: result Object

Examples:

url = blogurl + "/xmlrpc.php";
myRequest = y.rest(url);
myRequest.contentType("text/xml");
results = myRequest.put('<?xml version="1.0" encoding="UTF-8"?>' + postData.toString()).response;
query(key, value)

Adds a single query parameter.

Parameters:

  • key <String>
  • value <String>

Returns: request Object

Examples:

var formUrl = "http://example.com/form";
var resp = y.rest(formUrl).query("name","Tom").post().response;
query(hashmap)

Adds all the query parameters based on key-name hashmap.

Parameters:

  • hashmap <Object>

Returns: request Object

Examples:

var formUrl = "http://example.com/form";
var name = { "name": "John" };
var resp = y.rest(formUrl).query(name).post().response;
readTimeout(timeoutMillis)

Allows you to adjust the socket read timeout settings to wait longer (or shorter) for a response.

Parameters:

  • timeoutMillis <Number> The timeout value in milliseconds. The default value is 5000 milliseconds and the maximum value is 30000 milliseconds.

Examples:

var web_service_uri = "http://example.com/api";
var resp = y.rest(web_service_uri).readTimeout(10000).get().response;
timeout(timeoutMillis)

Specifies the request timeout in milliseconds. This is useful when you want to cancel requests that take longer than expected.

Parameters:

  • timeoutMillis <Number> The number of seconds before your request times out.

Returns: N/A

Examples:

// timeout after 500 milliseconds
y.rest('http://....').timeout(500).get();

response Global Object

The response global object allows you to determine how responses are handled and is also used to set the value to the data you’d like to return, as an E4X object, a JSON structure, or simply a string.

Object Description
object Contains the results of your execute script. Set this value to the data you’d like to return, as an E4X object, a JSON structure, or simply a string.

result Object

The result object contains the results of your execute script.

The table below lists the properties of the result object.

Property Description Data Type
response Get the response from the remote service. If the response content type is not application/json or text/xml then YQL provides a string. If JSON or XML is specified, the E4X representation of the data is returned. E4X object or string
headers The headers returned from the response. object
status The HTTP status code. string
timeout Indicates if the call timed out. boolean
url The URL used to make the request. string

Global Variables

The following global variables are available for use within the execute element of YQL Open Data Tables:

Variable Description
input

This global variable is available for each binding within the inputs element such as key, value, or map. For example, to call the first binding below you would use nameofid. var mycontent = nameofid;

Important

If the id name uses an illegal identifier, such as the use of hyphens, you must instead use the inputs global variable.

<inputs>
  <key id="nameofid" type="xs:string"
    paramType="path" default="my default key" required="true" />
  <value id="field-name" type="xs:string" paramType="variable"
  default="my default field" />
</inputs>

When a map binding is present in the Open Data Table, the global variable is present as a named hashtable. Each value provided in the YQL statement is set in the hashmap.

inputs

This global variable is an array that contains each binding within the inputs element, along with its value. For example, to call the second binding above, you would use inputs[fieldname]:

var mycontent =inputs['field-name'];

JavaScript and E4X Best Practices for YQL

The following is a series of best practices related to using JavaScript and E4X within the execute sub-element in YQL:

Paging Results

YQL handles paging of returned data differently depending on how you control paging within an Open Data Table definition. Let us consider the following example, followed be three paging element scenarios:

select * from table(10,100) where local.filter>4.0

  • No page element: If no paging element is provided, YQL assumes you want all data available to be returned at once. Any “remote” paging information provided on the select (10 being the offset and 100 being the count in our example), will be applied to all of the results before being processed by the remainder of the where clause. In our example above, the first 10 items will be discarded and only another 100 will be used, and execute will only be called once.
  • A paging element that only supports a variable number of results: If a paging element is provided that only supports a variable number of results (a single page with variable count), then the ``execute ``sub-element will only be called once, with the total number of elements needed in the variable representing the count. The offset will always be 0. In our example, the count will be 110, and the offset 0.
  • A paging element that supports both offset and count: If a paging element is provided that supports both offset and count, then the ``execute ``sub-element will be called for each “page” until it returns fewer results than the paging size. In this case, let’s assume the paging size is 10. The ``execute ``sub-element will be called up to 10 times, and expected to return 10 items each time. If fewer results are returned, paging will stop.

Note

In most cases, paging within the Open Data Table should match the paging capabilities of the underlying data source that the table is using. However, if the ``execute ``sub-element is adjusting the number of results coming back from a fully paging Web service or source, then there is usually no way to unify the “offset” of the page as set up in the Open Data Table with the destinations “offset”. You may need to declare your Open Data Table as only supporting a variable number of results in this situation.

Including Useful JavaScript Libraries

When writing your execute code, you may find the following JavaScript libraries useful:

OAuth:

y.include("http://oauth.googlecode.com/svn/code/javascript/oauth.js");
y.include("http://oauth.googlecode.com/svn/code/javascript/sha1.js");

Flickr:

y.include("http://blog.pipes.yahoo.net/wp-content/uploads/flickr.js");

MD5, SHA1, Base64, and other Utility Functions:

y.include("http://v8cgi.googlecode.com/svn/trunk/lib/util.js");

Using E4X within YQL

ECMAScript for XML (simply referred to as E4X) is a standard extension to JavaScript that provides native XML support. Here are some benefits to using E4X versus other formats, such as JSON:

  • Preserves all of the information in an XML document, such as namespaces and interleaved text elements. Since most web services return XML this is optimal.
  • You can use E4X selectors and filters to find and extract parts of XML structure.
  • The engine on which YQL is created natively supports E4X, allowing E4X-based data manipulation to be faster.
  • Supports XML literals, namespaces, and qualified names.

To learn more about E4X, refer to these sources online:

E4X Techniques

In addition to the resources above, the following tables provides a quick list of tips related to using E4X:

E4X Technique Notes Code Example
Creating XML literals - var xml = <root>hello</root>;
Substituting variables Use curly brackets {} to substitute variables. You can use this for E4X XML literals as well. var x = "text"; var y = <item>{x}</item>;
Adding sub-elements to an element When adding sub-elements to an element, include the root node for the element. item.node+=<subel></subel>;
You can add a sub-element to a node in a manner similar to adding sub-elements to an element. x.node += <sub></sub>;

This above code results in the following structure:

<node><sub></sub></node>

If you try to add a sub-element to a node without including the root node, you will simply append the element and create an XML list. x += <sub></sub>;

The above code results in the following structure:

<node><node><sub></sub>;

Assigning variably named elements Use substitution in order to create an element from a variable. var item = <{name}/>;
Assigning a value to an attribute - item.@["id"]=path[0];
Getting all the elements within a given element - var hs2 = el..*;
Getting specific objects within an object anywhere under a node - var hs2 = el..div
Getting the immediate H3 children of an element - h2 = el.h3;
Getting an attribute of an element -

h3 = el.h3.@id;

or

h3 = el.h3.@["id"];
Getting elements with a certain attribute - var alltn15divs = d..div.(@['id'] =="tn15content");
Getting the "class" attribute Use brackets to surround the "class" attribute. className =t.@['class'];
Getting a class as a string To get a class as a string, get its text object and the apply toString. var classString = className.text().toString()
Getting the name of a node Use localName() to get the name of a node. var nodeName = e4xnode.localName();

Note

When using E4X, note that you can use XML literals to insert XML “in-line,” which means, among other things, you do not need to use quotation marks:

var myXml = <foo />;

E4X and Namespaces

When working with E4X, you should know that E4X objects are namespace aware. This means that you must specify the namespace before you work with E4X objects within that namespace. The following example sets the default namespace:

``default xml namespace =’http://www.inktomi.com/‘; ``

After you specify a default namespace, all new XML objects will inherit that namespace unless you specify another namespace.

Caution

If you do not specify the namespace, elements will seem to be unavailable within the object as they reside in a different namespace.

Tip

To clear a namespace, simply specify a blank namespace:

``default xml namespace =’‘; ``

JavaScript Logging and Debugging

To get a better understanding of how your executions are behaving, you can log diagnostic and debugging information using the y.log statement along with the y.getDiagnostics element to keep track of things such as syntax errors or uncaught exceptions.

The following example logs “hello” along with a variable:

``y.log(“hello”); ``

y.log(somevariable);

Using y.log allows you to get a “dump” of data as it stands so that you can ensure, for example, that the right URLs are being created or responses returned.

The output of ``y.log ``goes into the YQL diagnostics element when the table is used in a select.

You can also use the follow JavaScript to get the diagnostics that have been created so far:

``var e4xObject = y.getDiagnostics(); ``

Executing JavaScript Globally

When the execute element is placed outside of the binding element, it is available globally within an Open Data Table and is usable across each of the table’s bindings. For example, in the following Open Data Table, the function arrayConcat is available to be used within the execute element for SELECT statements:

<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd" securityLevel="any">
    <meta>
        <author>liang</author>
        <description>execute block outside bindings</description>
        <documentationURL/>
    </meta>

      <execute><![CDATA[
         function arrayConcat(parents, children) {
         var family=parents.concat(children);
         return family;

       }
     ]]></execute>

    <bindings>

     <select itemPath="" produces="XML">
      <urls>
        <url env="all">http://fake.com</url>
      </urls>

      <inputs>
        <key id="mykey" type="xs:string" paramType="variable"
required="true" />
      </inputs>

     <execute><![CDATA[

        myparents=["Sister", "Brother"];
        mychildren=["Dad", "Mom"];
        var myfamily = arrayConcat(myparents,mychildren);
        response.object = <family>{myfamily}</family>;

     ]]></execute>

     </select>
    </bindings>
</table>

Making Asynchronous Calls with JavaScript Execute

The y.rest and y.query functions take callback functions as a parameter. The callback function is called either when y.rest or y.query completes a call, errors out, or times out. This callback mechanism allows for simultaneously handling and processing multiple REST calls.

The callback functions are just JavaScript functions that are passed the results of the REST call. When the callback function below is passed as a parameter to y.rest, the response is assigned to result.

The code example shows the basic syntax of using y.rest with a callback function. You can see how the callback can be implemented in Examples.

function callback(result) {
  // Do something with 'result'.
}
y.rest("http://example.com",callback);

// Wait for all the pending async operations to complete.
y.sync(true);

Examples

y.rest

A single callback function can be used for multiple requests. The following example uses the same callback to log the URL and status of the response as well as the number of responses received.

<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd"securityLevel="any">
  <bindings>
    <meta>
      <sampleQuery> select * from {table}</sampleQuery>
    </meta>
    <select produces="XML">
      <inputs></inputs>
      <execute><![CDATA[
        // Return data from the first y.rest
        // request that completed successfully.
        var count = 0;
        var first = null;
        var mycallback = function(result) {
          count++;
          y.log("response received for url: " +result.url + ", Status: " +result.status + ", Error: " + result.error + ", Timeout: " + result.timeout +", callback count: " + count);
          if (first == null) {
            first = result.response;
          }
        }
        // Create three asynchronous 'y.rest' requests.
        y.rest("http://www.yahoo.com", mycallback).timeout(500).get();
        y.rest("http://movies.yahoo.com", mycallback).timeout(500).get();
        y.rest("http://news.yahoo.com", mycallback).timeout(500).get();
        // Wait until we have at least
        // one successful completion
        while (y.sync(false) != true && first == null) {
          continue;
        }
        // Set response to data received from the
        // first async request that completed
        // successfully.
        response.object = first;
      ]]></execute>
    </select>
  </bindings>
</table>
y.query

The syntax and usage for callbacks with y.query is nearly identical to that of y.rest. The callback function is passed as a parameter to y.query.

The code example below uses the callback to keep track of the successful calls, logs the results, and saves the first returned response.

<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd"securityLevel="any">
  <meta>
    <sampleQuery> select * from {table} where timeout=100</sampleQuery>
  </meta>
  <bindings>
    <select produces="XML">
      <inputs>
        <key id='timeout' type='xs:string' paramType='variable' required="true"/>
      </inputs>
      <execute><![CDATA[
        // Check how many queries timeout within user specified timeout value.
        var count = 0;
        var timeout_count = 0;
        var mycallback = function(result) {
          count++;
          y.log(count + ": Results received for query: " + result.query + ",Error: " + result.error + ", Timeout: " + result.timeout);
          if (result.timeout) {
            ++timeout_count;
          }
        }
        // Create 3 asynchronous y.query requests.
        y.query("select * from feednormalizer where url='http://news.yahoo.com/rss/topstories' and output='atom_1.0'", null,timeout, mycallback);
        y.query("select * from weather.forecast where location=90210", null,timeout, mycallback);
        y.query("select * from answers.search where query='cars' andcategory_id=2115500137 and type='resolved'", null, timeout, mycallback);

        // Wait for the callbacks to be completed.
        y.sync(true);
        // Return the last query that completed successfully
        response.object = "Number of queries that timed out: " + timeout_count +", timeout: " + timeout + " msec.";
      ]]></execute>
    </select>
  </bindings>
</table>

Preventing Lossy JSON Results

When returning JSON responses from an Open Data Table, you should be aware that YQL will transform the JSON to XML before returning results to the client. If the client using your table requests that YQL return a JSON result, your original JSON response may be altered during the conversion from XML back to JSON, resulting in “lossy” JSON. When you make REST calls from your Open Data Table that request JSON, your table will also be receiving “lossy” JSON. The following sections show how the user of your table and your table can prevent “lossy” JSON.

Clients Using Your Table

To prevent “lossy” JSON, the client application using your table and requesting JSON results must add jsonCompat=new to the query string that is appended to one of the YQL Web service URLs. See JSON-to-JSON Transformation for details and usage examples. The YQL Query Parameters also lists and describes the jsonCompat query parameter.

Making REST Calls from Your Table

When making REST calls to fetch JSON from an Open Data Table, you must use the jsonCompat("new") function to get “lossless” JSON as well. In the example table below, the REST call to the Gowalla API chains the jsonCompat method with the get method to get “lossless” JSON. The client application using this table and requesting a JSON response must, however, also append jsonCompat=new to the YQL Web service URL to receive the “lossless” JSON.

...
  <bindings>
    <select>
      <urls>
        <url>http://api.gowalla.com/users/{id}</url>
      </urls>
      <inputs>
        <key id="id" type="xs:string" paramType="path" required="true"/>
        <key id="api_key" type="xs:string" paramType="variable" required="true"/>
      </inputs>
      <execute><![CDATA[
        var req =   request.header('X-Gowalla-API-Key', api_key).header('Accept', 'application/json');
        response.object =  req.jsonCompat("new").get().response;
      ]]>
      </execute>
    </select>
  </bindings>
...

Examples of Open Data Tables With JavaScript

The following Open Data Tables provide a few examples of YQL’s abilities:

Hello World Table

The following Open Data Table allows you to search a fictional table in which “a” is the path and “b” is the term.

This table showcases the following:

  • use of E4X to form the response
<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd">
  <meta>
    <sampleQuery>select * from {table} where a='cat' and b='dog';</sampleQuery>
  </meta>
  <bindings>
    <select itemPath="" produces="XML">
      <urls>
        <url>http://fake.url/{a}</url>
      </urls>
      <inputs>
        <key id='a' type='xs:string' paramType='path' required="true" />
        <key id='b' type='xs:string' paramType='variable' required="true" />
      </inputs>
      <execute><![CDATA[
          // Your javascript goes here. We will run it on our servers
          response.object = <item>
                   <url>{request.url}</url>
                   <a>{a}</a>
                   <b>{b}</b>
                </item>;
      ]]></execute>
    </select>
  </bindings>
</table>

Run this example in the YQL console.

Yahoo Messenger Status

The following Open Data Table allows you to see the status of a Yahoo Messenger user.

The table showcases the following:

  • use of JavaScript to check Yahoo Messenger status
  • use of E4X to form the response
<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd">
  <meta>
    <sampleQuery>select * from {table} where u='sample_id';</sampleQuery>
  </meta>
  <bindings>
    <select itemPath="" produces="XML">
      <urls>
        <url>http://opi.yahoo.com/online?m=t</url>
      </urls>
      <inputs>
        <key id='u' type='xs:string' paramType='query' required="true" />
      </inputs>
      <execute><![CDATA[

          //get plain text back from OPI endpoint
          rawStatus = request.get().response;

          //check if users is not offline
          if (!rawStatus.match("NOT ONLINE")) {
              status = "online";
          } else {
              status = "offline";
          }

          //return results as XML using e4x
          response.object =
          <messengerstatus>
             <yahoo_id>{u}</yahoo_id>
             <status>{status}</status>
          </messengerstatus>;
      ]]></execute>
    </select>
  </bindings>
</table>

Run this example in the YQL console.

OAuth Signed Request to Netflix

The following Open Data Table allows you to make a two-legged OAuth signed request to Netflix. It performs a search on the Netflix catalog for specific titles.

This table showcases the following:

  • access an authenticating API that requires signatures
  • use an external JavaScript library
<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd" https="true">
    <meta>
        <author>Paul Donnelly</author>
        <documentationURL>http://developer.netflix.com/docs/REST_API_Reference#0_52696</documentationURL>
    </meta>
      <bindings>
        <select itemPath="" produces="XML" >
          <urls>
            <url env="all">http://api.netflix.com/catalog/titles/</url>
          </urls>
         <paging model="offset">
           <start id="start_index" default="0" />
           <pagesize id="max_results" max="100" />
           <total default="10" />
         </paging>
          <inputs>
                <key id="term" type="xs:string" paramType="query" required="true" />
                <key id="ck" type="xs:string" paramType="variable" required="true" />
                <key id="cks" type="xs:string" paramType="variable" required="true" />
          </inputs>
          <execute><![CDATA[
// Include the OAuth libraries from oauth.net
y.include("http://oauth.googlecode.com/svn/code/javascript/oauth.js");
y.include("http://oauth.googlecode.com/svn/code/javascript/sha1.js");

// Collect all the parameters
var encodedurl = request.url;
var accessor = { consumerSecret: cks, tokenSecret: ""};
var message = { action: encodedurl, method: "GET", parameters: [["oauth_consumer_key",ck],["oauth_version","1.0"]]};
OAuth.setTimestampAndNonce(message);

// Sign the request
OAuth.SignatureMethod.sign(message, accessor);

try {
   // get the content from service along with the OAuth header, and return the result back out
   response.object = request.contentType('application/xml').header("Authorization", OAuth.getAuthorizationHeader("netflix.com", message.parameters)).get().response;
} catch(err) {
   response.object = {'result':'failure', 'error': err};
}
          ]]></execute>
        </select>
       </bindings>
</table>

Run this example in the YQL console.

Request for a Flickr frob

The following Open Data Table example returns the frob, which is analogous to the request token in OAuth.

This table showcases the following:

  • access an authenticating API that requires signatures
  • use an external JavaScript library
  • sign a request, then send the request using y.rest
  • require the HTTPS protocol (since private keys are being transmitted)
<?xml version="1.0" encoding="UTF-8" ?>
// https="true" ensures that only HTTPs connections are allowed
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd" https="true">
  <meta>
    <sampleQuery> select * from {table}</sampleQuery>
  </meta>
  <bindings>
    <select itemPath="rsp" produces="XML">
      <urls>
        <url>http://api.flickr.com/services/rest/</url>
      </urls>
      <inputs>
        <key id='method' type='xs:string' paramType='variable' const="true" default="flickr.auth.getFrob" />
        <key id='api_key' type='xs:string' paramType='variable' required="true" />
        <key id='secret' type='xs:string' paramType='variable' required="true" />
      </inputs>
      <execute><![CDATA[
        // Include the flickr signing library
        y.include("http://www.yqlblog.net/samples/flickr.js");
        // GET the flickr result using a signed url
        var fs = new flickrSigner(api_key,secret);
        response.object = y.rest(fs.createUrl({method:method, format:""})).get().response();
      ]]></execute>
    </select>
  </bindings>
</table>

Run this example in the YQL console.

CSS Selector for HTML

The following Open Data Table allows you to filter HTML using CSS selectors.

This table showcases the following:

  • importing external JavaScript utility functions
  • calling a YQL query within execute
<?xml version="1.0" encoding="UTF-8" ?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd">
  <meta>
    <sampleQuery>select * from {table} where url="www.yahoo.com" and css="#news a"</sampleQuery>
  </meta>
  <bindings>
  <select itemPath="" produces="XML">
    <urls>
      <url></url>
    </urls>
    <inputs>
      <key id="url" type="xs:string" paramType="variable" required="true" />
      <key id="css" type="xs:string" paramType="variable" />
    </inputs>
      <execute><![CDATA[
        //include css to xpath convert function
        y.include("http://james.padolsey.com/scripts/javascript/css2xpath.js");
        var query = null;
        if (css) {
          var xpath = CSS2XPATH(css);
          y.log("xpath "+xpath);
          query = y.query("select * from html where url=@url and xpath=\""+xpath+"\"",{url:url});
        } else {
         query = y.query("select * from html where url=@url",{url:url});
        }
        response.object = query.results;
      ]]></execute>
    </select>
  </bindings>
</table>

Run this example in the YQL console.

Execution Rate Limits

The following rate limits apply to executions within Open Data Tables:

Item Limit
Total Units of Execution 50 million
Total Time for Execution 30 seconds
Total Stack Depth 100 levels
Total Number of Concurrent YQL Queries 5 concurrent queries
Total Number of Objects Created via new 1 million objects
Total Number of Elements per E4X Object 1 million elements per E4X object

What is a unit of execution?

A unit can be any usage of memory or instruction. For example, if a specific operation is only used twice within an execute script, that would sum up to 2 units:

f(units) = f(operation1) + f(operation2)

Note

The total number of units allowed per operation can be lower than the maximum allowed if the script contains other operations which count towards the total units.

The following unit costs apply toward execution rate limits:

Unit Cost
y.query() 2000 units
Methods of the y global object (such as y.log() and y.rest()) 1000 units
String concatenation Length of the string being concatenated (1 unit per character)
Operation of an object created via new 500 units per operation
Addition of an element 50 units

The following example calculates the number of units needed when adding two XML trees that each contain 10 elements:

(10 elements + 10 elements) * 50 unit cost per element = 1000 units.