Remote and Local Processing

When YQL runs a SELECT statement, it accesses a back-end datasource, typically by calling a Web service. Remote filters and limits are implemented by the back-end Web service. Local processing (including local filters and limits) is performed by the YQL Web Service on the data it fetches from the back-end Web service. As shown by the following example, whether an operation is remote or local affects the data returned to the application that calls the SELECT statement.

Example With Remote and Local Steps

The following SELECT statement gets place information matching the string "San Francisco" from the geo.places table:

The steps that follow show the order in which YQL processes the remote and local parts of the SELECT statement:

  1. YQL calls Yahoo GeoPlanet API, the Web service behind the geo.places table, at the following URL: Francisco);start=0;count=50?appid=[yourappidhere]

    By calling this URL, YQL gets the first 50 items that match the SELECT statement's remote filter: q(San Francisco). The q element is the text input key for the geo.places table. Although the remote filter in the SELECT is set to 100, to improve efficiency, YQL only fetches 50 items each time it calls the Web service.

  2. On the 100 items it retrieved from the GeoPlanet API, YQL applies the local filter: lang in ("es"). This filter selects an item if the value lang field (column) is "es". From the set of the 50 items, YQL finds a subset of items that match the local filter.
  3. YQL checks that the items retrieved contain the lang, woeid, name, and country.content fields, which are specified after the SELECT keyword.
  4. YQL calls BOSS again, incrementing the start parameter to 50, to get the next 50 rows: 0;Francisco);start=50;count=50?appid=[yourappidhere]

    On this second set of 50 items, YQL applies the local filter, and again only finds a subset of items matching the local filter.

  5. On the second set of items from the GeoPlanet API, YQL applies the local filter and again verifies that the lang, woeid, name, and country.content fields, exist. Let's suppose that YQL has already found 5 matches from the two sets of results. Because the local limit of 5 has been reached, YQL does not call the GeoPlanet API again.
  6. YQL pipes the 5 items to the sort function, ordering the data by the country field.
  7. YQL returns 5 rows (containing just the lang, woeid, name, and country.content fields) to the calling Web application.

Summary of Remote and Local Controls

The following table identifies whether an element in the SELECT statement is processed locally or remotely by YQL.

Syntax Element in SELECT Local or Remote Section With Details
Columns or asterisk after the SELECT keyword. Local Specifying the Elements Returned (Projection)
Remote limit and offset, indicated by integers in parentheses after the table name. Remote Remote Limits
Remote filter expression in the WHERE clause. The allowed operators are the equal sign or IN. The value compared is an input key for the back-end datasource. Remote Remote Filters
Local filter expression in the WHERE clause. Various operators are allowed, including LIKE and MATCH. The value compared is a field (column) in the data returned by the query. Local Local Filters
LIMIT and OFFSET keywords after the WHERE clause. Local Local Limits
Sort and other functions after the pipe (|) symbol. Local Sort and Other Functions

Table of Contents