I'm getting reports that my XDA forum thread parser has become unreliable the last couple of days. I've started looking into it yesterday when the first report came in, and all I can see is that even simple operations fail.
What I've seen so far is that the four pages of the XDA forum thread are parsed correctly. But if I use a simple "Reverse" somewhere along the Pipe I can see that about halve the messages are dropped. The second "Unique: item.content" would produces 43 unique messages and if I feed that output into a Reverse operator I would only see 19 messages coming out on the other end. The exact number of messages, in and out, would differ with each run.
Is this a typical sign of performance problems in the Yahoo Pipes engine?
I've now "solved" it by reversing the order in which the pages of the XDA threads are parsed. The last page of the thread is now parsed first, so messages on that page are more likely to make it all the way to the end of the pipe.
While I'm writing this message the performance seems to have gone back to normal again. Any insight on what might be causing these kind of problems would be really helpful.