Could be a cool YQL app if there is no "Redirected to a robots.txt restricted URL" on many urls
Just saw a similar issue from another post with the suggestion that the source has disallowed bots from indexing a url. In the example above, I did check the robots.txt from google and the url is not on the disallow list. Similar situation with this url
An error caused the engine to disallow robots for this domain
the source's robots policy (finance.yahoo.com/robots.txt) do not have any listed restriction on the url I requested.
Seems like everything is running fine now. I have not hit a robots.txt error in the past 2 days with no code change on my part.
I see the potential with YQL and would like to write more apps using the service. However, stability is a main issue and I hope the YQL team can make Qos a top priority. At this point, I think YQL is great for experimental projects, not ready for prime time yet.