New YQL release makes select* from internet even more convenient

We've launched some pretty big updates during the last few months, <execute> for running server side JS code in your Open Data Tables, and new verbs for modifying and updating data on the web. Our release today consists of a lot of features (and bug fixes) that have been identified by developers on twitter, on our forums, or by our colleagues and ourselves - we use YQL all the time and certain things annoy us too!

The two major new feature additions are: the SET verb and the yql.storage tables.

SET enables you to configure static variables, such as API keys, secrets, and other required values, independently of YQL statements and web service calls. Why is this so useful? Consider the following example which uses the Guardian content search Open Data Table:

select * from guardian.content.search where api_key="1234567890" and q='environment'

Currently, every time I use the Guardian tables I need to send my API key - which really doesn't change very much. With set, I can just define it once in an environment file and forget about it.

set api_key="1234567890" on guardian;

And then add that environment file (which can contain many set and use statements) to my normal YQL request. Now I no longer need to specify that key in my queries on the Guardian API:

select * from guardian.content.search where query="environment"

Voilà! Although this separates these keys from the application logic I’m trying to accomplish with the select statement, I still need to put that key onto an environment file and host it on a server somewhere, e.g. http://hostingdomain.com/myenv.txt...

Enter the yql.storage and yql.storage.admin tables, which allow you to store and work with data using YQL itself. You can interact with these Open Data Tables using the same keywords used with other YQL statements: namely SELECT, INSERT, UPDATE, and DELETE. This means that developers of open data tables can easily share tables, environments and JavaScript with anyone, without having to host or expose these definitions in a public server or in github. What's more, data that you store for use with YQL is hosted in Yahoo!'s Sherpa cloud storage infrastructure - a scalable, elastic, and geographically distributed storage system - so it's quick no matter where you use YQL in the world.

Creating a hosted environment file is as easy as a single INSERT command in the console:

insert into yql.storage.admin (url) values ("http://hostingdomain.com/myenv.txt")

This returns three different access keys, or store urls:

<inserted>
   <execute>store://4cd56a99-7xxx-4600-ade1-3bf7f5a11193</execute>
   <select>store://08fd2c74-dxxx-48c4-9ee1-3bf7e5a1ec92</select>
   <update>store://3cc85a99-6xxx-4600-ade1-f7f83ecc4b83</update>
</inserted>

And to use it, all I need to do is use the "execute" store URL that's returned by the INSERT, e.g.,



https://developer.yahoo.com/yql/console/?q=select%20*%20from%20guardian.content.search%20where%20q%3D%27environment%27&env=http%3A%2F%2Fdatatables.org%2Falltables.env&env=store://4cd56a99-7xxx-4600-ade1-3bf7f5a11193

This works for tables, environments, JS libraries, and so on.

Between them, these two features enable a more secure and convenient method for accessing data using YQL queries. To learn more, take a look at our documentation on set and yql.storage. In addition, here are some the other treats on the smorgasbord for table developers:

  • multiple "env" files. You can now add multiple "env" query parameters to import many environment files at once. This is really useful when combined with the set and store features above.
  • crypto functions in <execute>. We've all used those nice JS libraries for sha1 and md5, but we've exposed a set of crypto functions in the new y.crypto global object to do it faster and more easily.
  • debug mode for table development. YQL's caches are pretty aggressive at times, and when you are trying to iteratively develop a table they can get in the way. It's also difficult to see exactly what YQL sent to a web source and what was sent back. With this release, you can add debug=true to your YQL web service call and network activity is captured and our caches get out of the way.
  • key aliasing in open data tables. Sometimes those remote query string names just really aren't helpful. Aliasing enables table developers to use a more meaningful key name in the YQL statement.

For a complete list of fixes and other changes please take a look at the changelog which we always post with each release on our blog. Keep a look out over the next few weeks as we explore some of these features more!

Jonathan Trevor
YQL lead