Yahoo! Pipes V2 engine now available

Pipes logoEditor's Notes: In a recent blog, A short adventure with Pipes, we mentioned the "forthcoming V2 Pipes engine." Well, it's no longer forthcoming: it's now available!

It's almost 3 years old, but the current Pipes engine was created quite quickly to see if the Pipes concept could succeed. It did. Wildly so.

Now the time has come to put something else in its place.

During the last year, the Pipes team has been developing a second-generation engine, which is used in the YQL product. It is much more stable and performs far better that the current engine. We've been working on migrating the old Pipes-specific functionality out of the v1 engine to this new engine.

Beta notes

We're at a beta stage now, where most of the big bugs are out and we need your help to find what we've missed. Please note:

  • Strict equality between the output of Pipes 1 and Pipes 2 is not a requirement. However, the Pipe should run and give output that makes sense.
    There will be bugs; this isn't ready for primetime.
  • The mechanism to run Pipes on the new engine is provided in confidence.
  • The Pipes 2 engine is a new baseline for new functionality, not the end goal.

Pipes 2 gives us a scalable platform to build features and capabilities that we've been unable to add to the old one, including: more sophisticated quota management, Pipe stat tracking, caching, and enhanced modules (like scripting).

Your feedback

What we'd like to get from you:

  • general feedback: It works the same, it doesn't, etc.
  • specific bugs: What doesn't work. Please give us the Pipe ID and, if you can pinpoint where things are going awry, all the better.

Once you find a bug (for example with a specific module), it will probably be common to many Pipes. Please avoid duplication and file the bug once.

Please post your feedback.

How to get started

  1. Log in to Pipes, then use the upgrade links on the pages or in the Pipes editor. Note: You can only upgrade your own Pipes.
  2. If you see a Pipe you want to run in V2 but do not own it, you can clone it and then upgrade it.

Next steps

On our end, we plan to implement auto-upgrade for all Pipes that work the same in both engines. After a number of months, we'll retire the old engine.

Thanks for trying out the V2 engine!