The problem exists in other domains, also. I thought that the cause could be the crawler being banned by robots.txt, but it turns out that when making a select from something below some path which is disallowed in robot.txt, the error message is different, as shown in the following examples.

A) Domain: www.greasespot.net
robots.tx:

User-agent: *
Disallow: /search

1) select * from html where url="http://www.greasespot.net"
Works fine.

2) select * from html where url="http://www.greasespot.net/search"

Returns "Error Retrieving Data from External Service"
<forbidden>robots.txt for that domain disallows crawling for that url</forbidden>

B)-- Nagesh

YQL Servers send the Client-ip header along with other headers as part of the request and it looks like the userscripts.org website doesnt like this header. A trivial curl request can reproduce the problem

CODE

curl -H "Client-ip: 68.180.220.12" http://userscripts.org

-- Nagesh

This seems to be the cause indeed. The same happens with lang-8.com. How did you know that?!
Anyway, this behaviour must date back a couple of weeks because I'd never had this problem before. This effectively breaks what for me is the main point of using YQL at all. Is there some way to tell Yahoo not to send the problematic header?

This is a standard header we pass to all the outgoing requests to let the websites know the identity of the client. Currently, there is no way to tell the YQL servers not to send the header but i believe that you have two options:
1) workaround the issue by proxying through an intermediary (request goes to an intermediate site which then forwards the request to userscripts w/o the header)
2) Bring this to the attention of the website and get this addressed. It looks like a bug if a header can cause it to return an internal error.

I had already thought of workaround 1, but this is somewhat beyond my knowledge and technical possibilities. The second option is not really an option, as "it is not scalable". I just cannot contact every server I want to read a page from and expect them to change their configuration just to please me, and that in a reasonable amount of time.

Then I found that this could be related to known a problem of Rails:

Client-IP Rails issue - Ruby Forum
http://www.ruby-forum.com/topic/168531

#322 Don't return 500 if Client-IP and X-Forwarded-For agree. - Ruby on Rails - rails
https://rails.lighthouseapp.com/projects/8994/tickets/322

In fact, the servers I observed the problem are:

userscripts-org: nginx + mod_rails/mod_rack
lang-8: Mongrel

What puzzles me a bit is that everything ceased to work suddenly a couple of weeks ago. Either Yahoo changed something in the way the request are sent, or everybody has updated to a version of Rails or of a Rails-based app that dislikes client-ip (or some combination of client-ip and other header/s).

We've been sending these headers for atleast the past 2 releases IIRC to adhere to our standards. Approximately since end of last month.

The proxy approach (1) would be to have an intermediate php (for example) hosted on a website which takes the URL as the parameter and then just returns the curl output of that URL. In the query one would have to refer to this php hosted url and give it the query parameter to fetch the data.
Unfortunately, there's no other solution that comes to mind right now.