YQL – The Swiss army knife of web services

Sebastian Spier is a software engineer for Meltwater Group. At work he spends a lot of time designing and analyzing APIs. Since 2010 he is actively contributing to the YQL Open Data Tables. He blogs at spier.hu.

YQL is the Swiss Army Knife of Web ServicesYQL is the Swiss Army Knife of Web ServicesPhoto by Jinho.Jung

The Yahoo Query Language (YQL) has been around since October 2008, and since then received a lot of positive coverage from inside and outside of Yahoo. While there are a couple of obvious purposes for which YQL can be used, there are others that are a bit more subtle.

This article assumes that you already know what YQL is and how to use it. (If not, the YQL Guide is a great start). Instead we focus on the areas in which YQL shines, although the technology might not always have been meant to be used for these purposes.

Less common YQL use cases

  1. Data Format Conversion
  2. Out of the box, YQL offers two data formats for each API that you interact with: XML and JSON. It does not matter if the host API really offers both formats, YQL will just do the conversion for you. This is quite practical but there are other cases in which YQL can also help you with format conversions.

    Say you need data from a website that publishes its data in CSV format - for example this stock data by Yahoo Finance. If you don’t want to deal with CSV directly in your application, you could convert the CSV to JSON with this query:

    SELECT * FROM csv WHERE url="http://finance.yahoo.com/d/quotes.csv?s=YHOO+GE&f=snl1d1t1ohgdr" (Ex. 1.1)

    One more example:
    The NBA is publishing the daily referee assignment on their website. Now you want to use this data to .. do something with it :) Again, you don’t want to mess with HTML in your application if not necessary, so you can have YQL convert it to XML for you:

    SELECT * FROM html
    WHERE url="http://www.nba.com/news/referee.html" AND xpath="//div[@class='dayContent']//td/p"
    (Ex. 1.2)

    Image from NBA WebsiteImage from NBA Website

    Image Credit: NBA.com

  3. The YQL Console as an Educational Tool
  4. Need to teach a computer science class at school or university? The YQL console can be a useful educational tool when experimenting with many sorts of online data, not just APIs. Think of it like your online terminal for data interaction.

    Help students experiment with XML data, and show them how to use XPath to select individual nodes in an XML document:

    SELECT * FROM xml
    WHERE url='http://rss.news.yahoo.com/rss/topstories' AND itemPath="//item/title"
    (Ex. 2.1)

    Teaching something about HTTP headers and how they impact the way the web works? Then how about this query as an easy way to show how the HTTP headers look like:

    SELECT headers FROM data.headers
    WHERE url="https://developer.yahoo.com/yql/guide/"
    (Ex. 2.2)

    Want to learn CSS selectors e.g. for working with tools like jQuery? Even here YQL can help you out with queries like this:

    SELECT * FROM data.html.cssselect
    WHERE url="http://www.w3.org/" AND css="ul.theme"
    (Ex. 2.3)

  5. Test your API Design
  6. How can YQL help you to check the design of your own API for flaws? YQL comes with its own description syntax that you need to use when you want to query your API with YQL. Such a description is called an Open Data Table definition but for the sake of simplicity people commonly just call it YQL Table.

    As it is possible to express almost any API as a YQL Table, you can assume that the YQL Table description syntax follows some of the best practises in API design. That means if you create a YQL Table for your own API, and you experience major difficulties with that, then this is an indication that you have designed your API and URL syntax in a non-standard way. This does not necessarily mean that you have to change your API all over but it should at least get your thinking. If you are interested in this topic, you may also want to read my more detailed post.

    Also analyzing the YQL tables for APIs created by others can be a good way to learn more about API design in general.

  7. The meta aspects of YQL
  8. As a contributor to the YQL Open Data Tables, you normally just create a mapping from an API to a YQL table and thats it. Still there are other kinds of YQL tables that you can create, that have a bit more of a “meta” flair to it.

    Example 1.2 above is actually such a meta use case, as you are essentially converting a simple HTML page into something that can be queried via YQL. So in a way you are creating a new API for data for which previously no API existed.

    Also if you mashup multiple APIs into one YQL table, you are really just creating a new meta API because it is potentially totally transparent to the user which APIs are called when he uses your YQL table.

    Even when using just a single API through YQL, the fact that you can perform queries that the original API does not support, also makes this a meta use case. Say the original API only lets you paginate 20 results at a time, so you need 5 individual API calls to get 100 results. With YQL you can get these 100 results in 1 request to YQL - as this request is internally parallelized into 5 requests to the original API. The nice thing about this: You can even have to know that this is happening, as it is all internal to YQL.

Conclusion

After reading this, I hope you are ready to explore YQL even further, and which better way than creating your own YQL tables right away? :) If so, I recommend you start with the section Creating YQL Open Data Tables in the YQL Guide. Nothing beats good examples, so also check out some of the tables created by others over at the Open Data Tables. Finally use the YQL Editor to try out your own tables, as this is the fastest way to debug them.

There are many more cases in which YQL can be useful that are outside of the scope of a normal API query. I would love to hear what other edge cases you use YQL for. Please post them and any further comments you may have in the comment section.