XPath Fetch Page returning Forbidden (403) inexplicably.


Looks like some of my pipes that use XPath Fetch Page have started returning 403 Forbidden, somewhat inexplicably. Unfortunately I have no idea when this started happening.

It does not seem to have anything to do with the user-agent or robots.txt file.

For instance, if I try a simple pipe:

XPath Fetch Page
URL: <http://mit.edu/>
Extract using XPath: [blank]
Use HTML5 parser: [not checked]
Emit items as string: [checked]

I get

Error fetching <http://mit.edu/.> Response: Forbidden (403)

(Earlier today, I saw the same problem to other well-known servers, like nytimes.com and yahoo.com. But those problems have gone away now.)


curl -A "Yahoo Pipes 1.0" <http://mit.edu/>

works just fine.

Any advice? How can I debug this? Thank you!

3 Replies

Recent Posts

in Pipes