0

YQL query returning all Twitter results as a single JSON-encoded string

This pipe calls the Twitter API via a YQL query:

http://pipes.yahoo.com/pipes/pipe.edit?_id=a28b29192f0e9a58e22de4328183d217

The call actually succeeds but the data returned is a single JSON-encoded string instead of a being a collection of individual items which can be manipulated by pipes. Is there anyway to get the data as items instead of one block of text?

by
8 Replies
  • Thanks for response. This actually didn't work for me, but it put me on track towards a solution, I hope. For whatever reason, the solution outlined in your reply resulted in just an even longer string with, now with the "results" element appended.

    I have ended up essentially constructing an entirely new object in the YQL table definition:

    var parsed = y.xparseJson(mycontent); var results = {}; for(var i = 0; i < parsed.length; i++) { results[parsed[i].id_str] = parsed[i]; }

    response.object = results;

    This at least give a set of individual "tweet" elements that could potentially be iterated over by Pipes. Unfortunately, Pipes itself is truncating the content of the individual elements. I can see in the YQL console that the elements all look fine, but when viewing the output in Pipes, most of the individual element attributes are not there. I'm going to try only passing the individual sub elements and see if that helps.

    0
  • Upon further review, the original code snippet:

    var parsed = '{"results": '  + resp +  '}';
    response.object=y.xparseJson(parsed);
    

    does produce individual items, but with a seemingly redundant sub-nodes:

    results
    + result
       + results
         + 0 
         + 1
         + 2
    

    Pipes see this a a single result unfortunately, so it doesn't make post processing from within Pipes any easier as there is only a single element. Ideally (I think) it would return:

    results
    + 0
    + 1
    + 2
    

    But anytime I try to assign response.object an Array() it complains that it expects and object not an array.

    0
  • Here's the YQL table I'm using:

    http://hapdaniel.pwp.blueyonder.co.uk/pdaniel/yql/twitter_user_timeline.xml

    In the Fetch data module I use a path of 'results.result.results'.

    0
  • That works, thanks. I guess I'll have to switch to using the Fetch module as you suggested.

    Still, I wish I understood why the YQL module itself couldn't be used to extract the data in useable form.

    0
  • With the YQL module you should use the Sub-element module with a value of 'item.results.results.results' (I think).

    0
  • I did find one solution: In my YQL table definition, I used:

    var parsed = '{"tweet": ' +  mycontent + '}';
    response.object=y.xparseJson(parsed); 
    

    Then changed the YQL select statement to:

    use my.table.xml as twitter; select tweet from twitter where...
    

    I could then use the Sub-element module with just "tweet" as a value and the output is now a list of elements instead of a single block. Unfortunately, somewhere along the line Pipes or YQL doesn't parse the json correctly, because most of the tweet elements are just dropped.

    I thought I had finally found a way to use the YQL module, but I may still be forced to go with the Fetch

    0
  • For the sake of completeness and anyone else who may have similar issues here is how I finally got around the issue of the YQL module not sending back the complete JSON. I only needed some of the JSON items, so I basically rebuilt it using only those items I was interested in.

    Parse the response json; loop through the elements; take out what I want and put into a new object; put that new object into a new array. Create a new outer "tweets" element, put the constructed array into the object and pass that back as the response.

    var resp = sendRequest({ 
        action: request.url, 
        method: 'GET', 
        accessor: accessor,
        parameters: null 
    });
    
    var parsed = y.xparseJson(resp);
    var content = []; // new array to hold the reconstructed content
    
    for(i = 0; i < parsed.length; i++) {
        var item = parsed[i];
    
        var tweet = {};
        tweet.text       = item.text;
        tweet.id         = item.id_str;
        tweet.created_at = item.created_at;
        tweet.in_reply_to_status_id = item.in_reply_to_status_id_str;
    
        if(item.retweeted_status != null)  {
            tweet.retweeted_status = {}
            tweet.retweeted_status.id = item.retweeted_status.id_str;
        }
    
        content.push(tweet);
    }
    
    // response.object can't take an Array directly; it has to be held by another object
    var parsed = y.xparseJson('{"tweets": ""}');    
    parsed.tweets = content;
    
    response.object = parsed;
    

    Using "select tweets from twitter" (or whatever you want to call the outer element in that last steps above) gives the output of the YQL mode a list of items I could work with.

    Maybe there's a better way to do it, but this worked for me.

    0

Recent Posts

in Pipes