0

YQL console giving forbidden error for somesites

Dear Dev's,

Why is my YQL console giving the error.
<forbidden>robots.txt for that domain disallows crawling for that url</forbidden>

for some "select from HTML" queries.
eg:select * from html where url="http://www.moneycontrol.com"

When i checked the robots.txt file ,it looks ok

http://www.moneycontrol.com/robots.txt

It (the robots file ) nowhere blocks yahoo ,or the URL as far as i can see.

Maybe i am wrong ,but can someone explain why this happens ....am interpreting he robots file wrongly?....

some geek from yahoo please help me on this ,please let me know if your other bots have been facing similar problems...

Regards,
Jynn

by
2 Replies
  • QUOTE (jynn @ Aug 2 2009, 10:17 PM) <{POST_SNAPBACK}>
    Dear Dev's,

    Why is my YQL console giving the error.
    <forbidden>robots.txt for that domain disallows crawling for that url</forbidden>

    for some "select from HTML" queries.
    eg:select * from html where url="http://www.moneycontrol.com"

    When i checked the robots.txt file ,it looks ok

    http://www.moneycontrol.com/robots.txt

    It (the robots file ) nowhere blocks yahoo ,or the URL as far as i can see.

    Maybe i am wrong ,but can someone explain why this happens ....am interpreting he robots file wrongly?....

    some geek from yahoo please help me on this ,please let me know if your other bots have been facing similar problems...

    Regards,
    Jynn


    This query works for me. Are you sure that's the URL thats causing the error?

    Jonathan
    0
  • QUOTE (Jonathan @ Aug 3 2009, 12:11 PM) <{POST_SNAPBACK}>
    This query works for me. Are you sure that's the URL thats causing the error?

    Jonathan

    Hi jonathan...
    strange....yesterday the exact thing was throwing forbidden error....maybe some network issue...
    My pipe is working again.....thanks anyway...
    0

Recent Posts

in YQL