Back to YQL Forum
0

YQL protects it's own REST URL's by robots.txt ?

Hi,
I have this query that's using the community table 'htmlstring' to fetch the output of another YQL query as a string: 
http://y.ahoo.it/RKZD7 
select * from htmlstring 
where url='http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20xml%20where%20url%3D%27http%3A%2F%2Fstackoverflow.com%2Ffeeds%2Fquestion%2F2671143%27%20and%20itemPath%3D%27feed.entry%27'

It fails with this error:

Exception: Redirected to a robots.txt restricted URL: http://query.yahooapis.com/v1/public/yql?q=<etc..>

I have another query that's using the open table 'xslt' that also uses the output of another YQL query, and it doesn't fail:
http://y.ahoo.it/1FJir
select * from xslt 
where url='http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20csv%20where%20url%3D%27https%3A//docs.google.com/spreadsheet/pub%3Fkey%3D0Au659FdpCliwdDZDaXBZOUE3X0hvdno4cDBRdHdLUnc%26single%3Dtrue%26gid%3D1%26output%3Dcsv%27' 
and stylesheet='https://raw.github.com/gist/1453419/97b90641c0f37ef755e842a3e85d6420818b12a7/gistfile1.xml'


What's the difference? Is there any reason why
1/ YQL query REST url's are blocked by Yahoo's robots.txt ?
2/ there's inconsistent behaviour when using different tables?
Thanks for your feedback on this!!
Regards,
Vic
PS. my usecase for the first example, is to wrap the XML output of the YQL REST URL as a string in JSONP, so I can transfer XML cross-domain without having to convert to and from JSON. Maybe there are other alternatives to do so?

by
Report Abuse
2 Replies
  • Hi,

    an extra note: I'm also getting Error 999 after a while when testing this stuff on the YQL console... It's debugging, so I'm not submitting queries at a high rate. But the fact that I'm using the output of one query as the input of another query is of course causing two calls to run almost simltaneously... Isn't that supported by YQL's rate limits?

    Regards,
    Vic
    Report Abuse
    0
  • Hi,

    I took a fresh view on the problem this morning and I noticed that I was rushing a bit too hard using the htmlstring table for wrapping *xml* data into a string. So I created a new table "xmlstring" (in the code it boiled down to changing the mime type) and that seems to work.

    http://y.ahoo.it/AlXl4
    use 'https://raw.github.com/vicmortelmans/yql-tables/master/data/xmlstring.xml' as xmlstring; select * from xmlstring where url='http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20xml%20where%20url%3D%27http%3A%2F%2Fstackoverflow.com%2Ffeeds%2Fquestion%2F2671143%27%20and%20itemPath%3D%27feed.entry%27'

    Here's some brief documentation for the table:
    https://docs.google.com/document/pub?id=1ATBRllWB72-ZirrcrEXHZsjGg2ublp1dRTqUjmi4ibA

    A pull request to yql/yql-tables has been submitted.

    Regards,
    Vic
    Report Abuse
    0
Ask a Question

Recent Posts

in YQL