Hubris aside (mine I mean), I'm probably not qualified to be the last word on this with regard to advising you on your project's overall direction, so hopefully others will chime in on this discussion...but that said, when I started with hadoop a few years ago and our project exceed the capabilities of direct-Java hadoop interfacing, we briefly entertained the notion of pipes and learned (to the best of my recollection) that pipes was officially losing its support within the hadoop community (it may have even been depricated although I don't remember for sure) and that streaming was being officially advocated as the replacement (unless I am grossly exaggerating the advice I received at the time). In fact, the primary reason I abandoned pipes wasn't so much that the technology itself wouldn't work, but that there was far less community support for it than for streaming, so whenever I got stuck I couldn't really ask for help (or I could ask but I would be inquiring a much smaller pool of potential responders, thus decreasing the scope of any response).
At any rate, I have been quite satisfied with streaming. Gotchas have included coding from a scratch an entire partially abstract C++ class family to offer a streaming interface to communicate between hadoop and underlying native processing, to say nothing of coding up binary/ascii conversion in a variety of formats to shuffle data across the i/o streams (for reasons I can't recall now, the binary streaming interface wasn't a good option for us at the time).
Still, all's well.
Best of luck.
On May 21, 2012, at 10:26 , Peter Cogan wrote:
> Hi Keith,
> Yeah - maybe that was my first mistake. Since I haven't had much luck getting pipes working I've started looking at streaming. I am still 'early in my hadoop efforts' so now is the time for big course corrections
> On 21 May 2012, at 18:11, Keith Wiley wrote:
>> Mind if I ask why you prefer pipes to streaming? I abandoned pipes early in my hadoop efforts and went all engines full on streaming.
>> On May 21, 2012, at 04:07 , Peter Cogan wrote:
>>> I am trying to install pipes on os x following the instructions here
Keith Wiley [EMAIL PROTECTED] keithwiley.com music.keithwiley.com
"The easy confidence with which I know another man's religion is folly teaches
me to suspect that my own is also."
-- Mark Twain