Remote and Local Processing

When YQL runs a SELECT statement, it accesses a back-end datasource, typically by calling a Web service. Remote filters and limits are implemented by the back-end Web service. Local processing (including local filters and limits) is performed by the YQL Web Service on the data it fetches from the back-end Web service. As shown by the following example, whether an operation is remote or local affects the data returned to the application that calls the SELECT statement.

Example With Remote and Local Steps

The following SELECT statement gets data about pizza restaurants from the search.web table:

The steps that follow show the order in which YQL processes the remote and local parts of the SELECT statement:

  1. YQL calls Yahoo! Search BOSS, the Web service behind the search.web table, at the following URL:

    http://boss.yahooapis.com/ysearch/web/v1/pizza?format=xml&start=0&count=50

    By calling this URL, YQL gets the first 50 items that match the SELECT statement's remote filter: query='pizza'. The query element is an input key for the search.web table. Although the remote filter in the SELECT is set to 500, to improve efficiency, YQL only fetches 50 items each time it calls BOSS.

  2. On the 50 items it retrieved from BOSS, YQL applies the local filter: ((title like 'Round%') or (abstract matches '.*about.*')). This filter selects an item if the title field (column) starts with Round or the abstract field contains about. In this example, from the set of 50 items, YQL finds 3 items that match the local filter.
  3. YQL checks that the items retrieved from BOSS contain the title , abstract, and url fields, which are specified after the SELECT keyword.
  4. YQL calls BOSS again, incrementing the start parameter to 50, to get the next 50 rows:

    http://boss.yahooapis.com/ysearch/web/v1/pizza?format=xml&start=50&count=50

    On this second set of 50 items, YQL applies the local filter, but finds no matches.

  5. YQL calls BOSS a third time to get the next 50 items:

    http://boss.yahooapis.com/ysearch/web/v1/pizza?format=xml&start=100&count=50

  6. On the third set of items from BOSS, YQL applies the local filter and verifies that the title, abstract, and url fields exist. This time, YQL finds 2 more matches, which brings the total number of matches to 5. Because the local limit of 5 has been reached, YQL does not call BOSS again.
  7. YQL pipes the 5 items to the sort function, ordering the data by the title field.
  8. YQL returns 5 rows (containing just the title, abstract, and url fields) to the calling Web application.

Summary of Remote and Local Controls

The following table identifies whether an element in the SELECT statement is processed locally or remotely by YQL.

Syntax Element in SELECT Local or Remote Section With Details
Columns or asterisk after the SELECT keyword. Local Specifying the Elements Returned (Projection)
Remote limit and offset, indicated by integers in parentheses after the table name. Remote Remote Limits
Remote filter expression in the WHERE clause. The allowed operators are the equal sign or IN. The value compared is an input key for the back-end datasource. Remote Remote Filters
Local filter expression in the WHERE clause. Various operators are allowed, including LIKE and MATCH. The value compared is a field (column) in the data returned by the query. Local Local Filters
LIMIT and OFFSET keywords after the WHERE clause. Local Local Limits
Sort and other functions after the pipe (|) symbol. Local Sort and Other Functions

Table of Contents