Issue with Gawker feeds for past few weeks

I've been having issues with some feeds that used to work great, but have not been working in the past few weeks (about beginning of April). I've isolated the issue to a number of Gawker feeds and have removed all of the other sources from the pipe (http://pipes.yahoo.com/pipes/pipe.edit?_id=54736d81389c9bb44a505863559fcb1a).

It appears the issue may be related to the full versions of the feeds (e.g., http://jalopnik.com/rss/vip), as opposed to the regular condensed ones (e.g., http://jalopnik.com/rss). I can add a Fetch Feed module to a new pipe using one of the condensed feeds and usually be able to retrieve results, however, as soon as I change to the vip feed, I frequently get Preview failed errors (cannotconnect, Server Hangup, or badly formed response). The server appears to be up because I'm able to consistently get recent results by putting the feed address directly into my browser.

I've also tried changing my pipe to use only the condensed feeds and still get errors.

Thanks for any inputs!

7 Replies
  • I'm guessing you need to authenticate to get those feeds?

  • @Lolo No authentication needed. You could put http://jalopnik.com/rss/vip into your browser address bar to test.

  • ok, I opened both with kate for a qualitative comparison, then tested the feed with the W3C validator.

    bascially my opinion is that neither feeds are well made, in particular due to all those<![CDATA[ [...] ]]>. The 'vip' feed simply has much more errors because there is more content, and that at some point the pipe processing hits a tolerance limit, is my guess.

    Have you tried to simply strip those mark ups (not the content though) and see if it works better?

  • Is there any way to strip out those mark-ups within Yahoo Pipes? Once the feed gets to a Fetch Feed module, Pipes parses out the components and the mark-ups don't appear, even if I look at the item.description source. Today, it seems I am able to add the vip feed to a new pipe and get results. As soon as I add an operator module such as filter with a simple filter (e.g., block where item.title contains "foo"), I get the server errors. So essentially, I cannot alter the feed, which defeats the purpose of adding it to a pipe.

  • yes, use a regex. The reason is that the processing of the pipe works with an xml structure, and those tags get in the way.

    try the following in a regex module: change (?:\<!\[CDATA\[ ?\[)|(?:\] ?\]\]\>) to (nothing) [x] [x] [ ] [ ]

    I haven't tried it 'cause simply fetching the feed (hence all 3 fetch feed modules unlinked) resulted in preview failed :/

  • If I'm not mistaken, adding the feed to a Fetch Feed module parses out the feed into items and its sub-components. It appears to also process the contents and strip out those mark-ups. (You could test this by adding http://jalopnik.com/rss/vip to a new pipe and viewing the debugger results or see http://pipes.yahoo.com/pipes/pipe.edit?_id=572f801f8cbb1b6a44b53161ebc88ca9) As such, I shouldn't need to add a Regex module to strip out the mark-ups as suggested. Even if I wanted to, as soon as I add an operator module to the test pipe, the server errors appear, which leads me to think that this is something on the YQL side because I'm able to view the unaltered feed, but unable to alter it with an operator module. I'm wondering if the server resource allocation per pipe has changed in the past month.

    I appreciate your suggestions. Any other thoughts?

  • what I meant was, those special tags are used by pipes. so no, they do not appear when the feed is fetched, but they're still there. the trouble is the regex limit of 17500 characters, so I used a string replace instead.

    If I put item.content as input so I filter out all CDATA tags, it looses all data and all but first sub-item is lost. If I put item.description, it fails, which tends to enforce my opinion.


Recent Posts

in Pipes