0

Trying to add a unique guid to each item in rss feed

I have an rss feed that doesn't contain unique guid's per item.

I'm trying to use a wordpress addon to import items in an rss feed to posts but the addon only posts the first item in the feed because it assumes there all the same because they all have the same guid. The addon uses the guid to determine if an item in the feed has been posted already so it doesn't post the same thing twice.

So what I need to do is use yahoo pipes to create a unique guid per item in the feed so I can use the wordpress addon.

My rss feed has my login details so I cant share the original feed but here is the feed structure.

<item>
<title>Title here</title>
<category>Catagory Here</category>
<description>Description Here</description>
<pubDate>Wed, 05 Feb 2014 12:00:00 +0000</pubDate>
</item>

How would I go about this. I'm new to Pipes so please bare with me.

by
10 Replies
  • Have a look at this dummy pipe...

    Pipes: FOR FORUM - GUID example http://pipes.yahoo.com/pipes/pipe.info?_id=e956194ba824b095c6a92f6c56eb2402

    ...in the editor. It's unfinished so I didn't bother to make

    item.link

    It uses apple, banana, cherry as item.title

    ===

    String Builder + String Tokenizer modules They us BR tags to create the items.

    Rename module It creates item.title

    Regex module It creates a dummy item.pubDate so you can see how things work.

    It could be item.TEMP is unique but let's assume it's not. So...

    Regex module It creates item.UNIQUE_THING which is item.title plus item.pubDate

    ===

    If the items aren't unique (say you get lazy and only use item.title) - items will be silently "filtered".

    Consider these two items with identical item.title

    • "Apple" Monday post is published on Monday
    • "Apple" Friday post is published on Friday

    The result is

    • "Apple" is published on Monday
    • On Friday the Monday post is rewritten with Friday's data.

    Pipes will not make a Friday item. Instead pipes will take the Friday data and put it into Monday item overwriting the original data.

    0
  • Hi,

    Make the assumption all items are unique (if not, you can filter in pipes). With that assumption, you just want to put distinct strings in the guid field. To do that, you can just copy the title to guid (rename module, select item.title 'copy as' guid), or put utime in it (regex module), or a copy of the pubdate,...

    That's what you could do. What you should do, is to a link to the article in the guid field, and indicate if it's a permaLink or not: that is the role of the guid: http://validator.w3.org/feed/docs/rss2.html#ltguidgtSubelementOfLtitemgt

    So if the link is present in the description for example, you should copy it as guid, then use regex to leave only the link as the guid value. Then, still using the regex for example, create the field guid.isPermaLink and set it to 'true'.

    Have fun discovering Pipes!

    0
  • Ok so I created a rename module to copy the title to the guid that is working fine Only if I use YQL to load feed. However if I use YQL to load feed my pubdate for each item disappears in the rss.

    If I use the Fetch Feed source pubdate stays intact but guid is only added to the first item...

    The first item has

    <guid isPermaLink="false"><Guid set properly as item title></guid>
    

    but all the rest of the items have...

    <guid isPermaLink="false"></guid>
    

    What am I missing?

    0
  • don't use YQL here, it's an overkill. fetch data or fetch feed, that's fine. However all fetching methods, used properly, should give you the same results.

    the <Guid set properly as item title> seems very wrong. you should obtain something like <guid isPermaLink="false">title of the i-th article of the feed</guid>, unless you put that here instead of your article title, in which case please don't as it's all kinds of wrong.

    From what I read, ie that your add-on read only the first article and that pipe does that too, I'd say neither are culprits, and you've got only one item in your feed.

    first, can't you link a dummy pipe without your login details? of another feed for example... even though I'd like to check this one as I don't believe very much there is no guid sub-item, and I suspect your feed is only one item long.

    ps: the norm specify that the guid is NOT mandatory, so if your really have more than one item in the feed, you should probably look for another wordpress add-on...

    0
  • I understand that the GUID is not mandatory for a feed to work. However this Wordpress addon needs the guid to be set in order to know if it processed the item in the feed or not. The wordpress addon makes a new post with the contents of the feed if the guid is missing or all items have the same guid the addon assumes the items are the same and skips duplicates.

    The Addon is not to blame. It's the original feed not containing the guid for the addon to work.

    I have added the feed to feedburner (To hide my user credentials) so I can now share my pipe hopefully this makes this easier. :)

    Here is my pipe...

    http://pipes.yahoo.com/pipes/pipe.info?_id=5bf217ec97bb26d680ed5f60c6350ded

    I need the guid to be unique per item So the title copy should do the trick.

    I also need the pubdate to continue working as the addon will backdate the posts depending on the pubdate and I want this.

    I really appreciate your help I know how hard it can be to troubleshoot without actually being able to see it, and for that I thank you for dealing with me the best you can.

    0
  • change guid to y:id (they're equivalent) and Pipes should produce guid instead.

    0
  • I did that but its not setting the unique guid.

    RSS output http://pipes.yahoo.com/pipes/pipe.run?_id=5bf217ec97bb26d680ed5f60c6350ded&_render=rss

    0
  • Do you know anything else I can try?

    0
  • Anybody?

    0
  • Hello,

    your problem was that your original feed already had a y:id value (an empty one) and a y:id.isPermaLink. Thus, trying to assign a dummy string to it was not to the linking of pipes it would seem...

    Instead, in this pipe, I built the guid in another sub-field (the guid here), and copied it whole to y:id afterwards. Also, I took out the spaces, in case of, but that might just be me being overcautious. Last word, when working on the guid, open your feed in a text editor to see the guid sub-field. However, the update of the pipe scheme is not instantaneous. Do not change the pipe right after a first change because the output is identical...

    0

Recent Posts

in Pipes