Using Hosted Storage with YQL¶

In this Chapter:

Introduction
Storing New Records
Using YQL to Read, Update, and Delete Records
Using Records within YQL

Introduction¶

YQL now provides two Open Data Tables, yql.storage and yql.storage.admin, that allow you to store and work with data using YQL itself. These default tables are available in the YQL console under the “yql” category. You interact with these Open Data Tables using the same keywords used with other YQL statements: namely SELECT, INSERT, UPDATE, and DELETE.

About YQL Hosted Storage¶

Data that you store for use with YQL is hosted in Yahoo’s Sherpa cloud storage infrastructure. It is a scalable, elastic, and geographically distributed storage system used by many Yahoo services. Benefits of using Sherpa include:

low latency that can handle a large number of concurrent requests.
automated load balancing and failover to reduce operational complexity.
high availability and fault tolerance.

For more information on Sherpa, refer to Moving to the Cloud” on the Yahoo Developer Network (YDN) Blog.

Storage Limits and Requirements¶

The following storage limits and requirements apply to records stored using YQL:

Size Limit: You can have up to 1000 records, with each record being up to 100KB.
Retention Limit: Records not read, updated, or executed at least once every thirty days may be removed.
Record Format: Records must be in a text-based format. Examples include JavaScript code, XML files, Open Data Tables, or YQL environment files.
Authentication for New Records: All connections to yql.storage.admin must be authorized using two-legged OAuth. Alternatively, you can create new records using the YQL console. However, connections to ``yql.storage ``do not require authentication.

Storing New Records¶

You create new records in three ways using the ``yql.storage.admin `` Open Data Table:

value: Create a record and insert text into it.
url: Create a record and insert content from an existing URL.
name, url: Create a record with an execute key that you name and insert content from an existing URL.

Note

Any data you store within a record cannot exceed 100KB.

Upon creating a new record, YQL responds with three access keys, each of which serves a different function, as seen in the following response snippet:

...
    <results>
        <inserted>
            <execute>store://35ad2c72-e353-41c4-9d21-4a7e5a1ec92</execute>
            <select>store://08fd2c74-d393-48c4-9ee1-3bf7e5a1ec92</select>
            <update>store://3cc85a99-6a89-4600-ade1-f7f83ecc4b83</update>
        </inserted>
    </results>
...

There are three access keys associated with each record in storage, each starting with store://:

execute: This access key allows the value of the record to be used in a YQL statement containing use statements or referring to env files. You can share this access key with other developers, who can use the Open Data Table or environment file but will be unable to “read” the table definition or environment details.
select: This access key allows the value of the record to be used in YQL SELECT statements.
update: This access key allows the value of the record to be used in YQL UPDATE and DELETE statements.

Note

Knowing the execute, select, or update access key is sufficient to perform those operations on the stored record. You should only share with others the access key that grants the desired permissions.

Storing a New Record using Text¶

To create a new text record in storage for YQL, use the following statement format:

``insert into yql.storage.admin (value) values (“example text content”) ``

Storing a New Record using Data from an URL¶

To copy the contents of an URL, such as an environment file or Open Data Tables, into a new record for YQL, use the following statement format:

``insert into yql.storage.admin (url) values (“http://hostingdomain.com/mytable.xml”) ``

Storing a New Named Record using Data from an URL¶

To copy the contents of an URL, such as an environment file or Open Data Tables, into a new record with a custom ``execute ``access key, use the following statement format:

``insert into yql.storage.admin (name,url) values (“newrecord”,”http://hostingdomain.com/mytable.xml”) ``

When you create a record using the above format, the execute access key uses the name (newrecord) and top-level domain of the URL (hostingdomain.com) that you supply, as seen in the following response snippet:

...
    <results>
        <inserted>
            <execute>store://hostingdomain.com/newrecord</execute>
            <select>store://08fd2c74-d393-48c4-9ee1-3bf7e5a1ec92</select>
            <update>store://3cc85a99-6a89-4600-ade1-f7f83ecc4b83</update>
        </inserted>
    </results>
...

Tip

Using the SET keyword in conjunction with a storage record, you can set and hide values, such as passwords, API keys, and other required values independently of YQL statements and API calls . For more information, refer to Setting Key Values for Open Data Tables.

Using YQL to Read, Update, and Delete Records¶

Once you create a record using the yql.storage.admin Open Data Table, you can use the SELECT, UPDATE, and DELETE keywords with yql.storage to access, modify, or delete records, respectively.

Accessing Records using YQL¶

To read or access a record using YQL, use the following statement format that contains the SELECT keyword:

``select * from yql.storage where name=”store://08fd2c74-d393-48c4-9ee1-3bf7e5a1ec92” ``

Use the select access key that YQL provided to you upon creating the record. The YQL response contains a results element similar to the following:

...
    <results>
        <value>example test content</value>
    </results>
...

Deleting Records using YQL¶

Use the DELETE keyword in YQL to delete a record, similar to the following:

``delete from yql.storage where name=”store://3cc85a99-6a89-4600-ade1-f7f83ecc4b83” ``

Use the update access key that YQL provided to you upon creating the record. The YQL response contains a results element similar to the following:

...
    <results>
        <success>store://3cc85a99-6a89-4600-ade1-f7f83ecc4b83 deleted</success>
    </results>
...

Updating Records using YQL¶

Use the UPDATE keyword in YQL to modify a record, similar to the following:

``update yql.storage set value=”new value” where name=”store://3cc85a99-6a89-4600-ade1-f7f83ecc4b83” ``

Use the update access key that YQL provided to you upon creating the record. The YQL response contains a results element similar to the following:

...
    <results>
        <success>Updated store://3cc85a99-6a89-4600-ade1-f7f83ecc4b83</success>
    </results>
...

Note

You can only update the contents of a named execute record to a URL on the same URL domain.

Using Records within YQL¶

If your record contains an environment file, an Open Data Table, or JavaScript, YQL can “execute” or run it in a read-only manner.

Using Hosted Environment Files¶

When a record contains a YQL environment file, use the corresponding execute access key in YQL calls as you would any environment file. The following example uses URL encoding for the execute access key and appends it to the YQL console URL:

https://developer.yahoo.com/yql/console?env=store%3A%2F%2Fopendatatables

Using Hosted YQL Open Data Tables¶

When a record contains an Open Data Table, use the corresponding execute access key in YQL calls as you would any Open Data File. In the following example, the USE keyword invokes the access key, which is then used as mytable:

use "store://35ad2c72-e353-41c4-9d21-4a7e5a1ec92" as mytable; select * from mytable;

Including Hosted JavaScript¶

When a record contains JavaScript, use the select access key in YQL to include it in an Open Data Table.

Note

You do not use the execute access key with JavaScript includes using y.include because parsing JavaScript is done in the same manner as reading it.

To do a JavaScript include, first insert the JavaScript into a record in the following manner:

insert into yql.storage.admin (name, url) values ('testjs', 'http://javarants.com/yql/test.js')

The response from YQL contains the following URLs:

<inserted>
   <execute>store://javarants.com/testjs</execute>
   <select>store://f7c24314-b0fd-488b-a67c-1d711ca3bc21</select>
   <update>store://077b02c0-b6e8-4923-90a4-9b8096ccdaec</update>
</inserted>

Use the select URL in the select element of the Open Data Table in the following manner:

// contains:
// var success = true;
y.include('store://f7c24314-b0fd-488b-a67c-1d711ca3bc21');
response.object = <success>{success}</success>;