I am doing a general poll on what are the most prevalent pain points that
people run into with Hadoop? These could be performance related (memory
usage, IO latencies), usage related or anything really.

The goal is to look for what areas this platform could benefit the most in
the near future.

Any feedback is much appreciated.

Thanks,
Kunal.

Interesting question.  Do you want to be asking those who use Hadoop ---
or those who find it too painful to use?

Regards,
Mike

From:   Kunaal <[EMAIL PROTECTED]>
To:     [EMAIL PROTECTED]
Date:   03/02/2012 11:23 AM
Subject:        Hadoop pain points?
Sent by:        [EMAIL PROTECTED]

I am doing a general poll on what are the most prevalent pain points that
people run into with Hadoop? These could be performance related (memory
usage, IO latencies), usage related or anything really.

The goal is to look for what areas this platform could benefit the most in
the near future.

Any feedback is much appreciated.

Thanks,
Kunal.

Lol!

Raj

>________________________________
> From: Mike Spreitzer <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]
>Sent: Friday, March 2, 2012 8:31 AM
>Subject: Re: Hadoop pain points?
>
>Interesting question.  Do you want to be asking those who use Hadoop ---
>or those who find it too painful to use?
>
>Regards,
>Mike
>
>
>
>From:   Kunaal <[EMAIL PROTECTED]>
>To:    [EMAIL PROTECTED]
>Date:   03/02/2012 11:23 AM
>Subject:        Hadoop pain points?
>Sent by:        [EMAIL PROTECTED]
>
>
>
>I am doing a general poll on what are the most prevalent pain points that
>people run into with Hadoop? These could be performance related (memory
>usage, IO latencies), usage related or anything really.
>
>The goal is to look for what areas this platform could benefit the most in
>the near future.
>
>Any feedback is much appreciated.
>
>Thanks,
>Kunal.
>
>
>
>

I am asking users who use Hadoop and love it, but would want to see it
improved in certain specific areas.

On Fri, Mar 2, 2012 at 8:31 AM, Mike Spreitzer <[EMAIL PROTECTED]> wrote:

> Interesting question.  Do you want to be asking those who use Hadoop ---
> or those who find it too painful to use?
>
> Regards,
> Mike
>
>
>
> From:   Kunaal <[EMAIL PROTECTED]>
> To:     [EMAIL PROTECTED]
> Date:   03/02/2012 11:23 AM
> Subject:        Hadoop pain points?
> Sent by:        [EMAIL PROTECTED]
>
>
>
> I am doing a general poll on what are the most prevalent pain points that
> people run into with Hadoop? These could be performance related (memory
> usage, IO latencies), usage related or anything really.
>
> The goal is to look for what areas this platform could benefit the most in
> the near future.
>
> Any feedback is much appreciated.
>
> Thanks,
> Kunal.
>
>
--
"What we are is the universe's gift to us.
What we become is our gift to the universe."

Hi Kunaal,

We have a recopilation of some of them here:
http://www.datasalt.com/2012/02/mapreduce-hadoop-problems/

Regards,
Iván

2012/3/2 Kunaal <[EMAIL PROTECTED]>

> I am asking users who use Hadoop and love it, but would want to see it
> improved in certain specific areas.
>
> On Fri, Mar 2, 2012 at 8:31 AM, Mike Spreitzer <[EMAIL PROTECTED]>
> wrote:
>
> > Interesting question.  Do you want to be asking those who use Hadoop ---
> > or those who find it too painful to use?
> >
> > Regards,
> > Mike
> >
> >
> >
> > From:   Kunaal <[EMAIL PROTECTED]>
> > To:     [EMAIL PROTECTED]
> > Date:   03/02/2012 11:23 AM
> > Subject:        Hadoop pain points?
> > Sent by:        [EMAIL PROTECTED]
> >
> >
> >
> > I am doing a general poll on what are the most prevalent pain points that
> > people run into with Hadoop? These could be performance related (memory
> > usage, IO latencies), usage related or anything really.
> >
> > The goal is to look for what areas this platform could benefit the most
> in
> > the near future.
> >
> > Any feedback is much appreciated.
> >
> > Thanks,
> > Kunal.
> >
> >
>
>
> --
> "What we are is the universe's gift to us.
> What we become is our gift to the universe."
>

--
Iván de Prado
CEO & Co-founder
www.datasalt.com

Since you ask about anything in general, when I forayed into using
Hadoop, my biggest pain was lack of documentation clarity and
completeness over the MR and DFS user APIs (and other little points).

It would be nice to have some work done to have one example or
semi-example for every single Input/OutputFormat, Mapper/Reducer
implementations, etc. added to the javadocs.

I believe examples and snippets help out a ton (tons more than
explaining just behavior) to new devs.

On Fri, Mar 2, 2012 at 9:45 PM, Kunaal <[EMAIL PROTECTED]> wrote:
> I am doing a general poll on what are the most prevalent pain points that
> people run into with Hadoop? These could be performance related (memory
> usage, IO latencies), usage related or anything really.
>
> The goal is to look for what areas this platform could benefit the most in
> the near future.
>
> Any feedback is much appreciated.
>
> Thanks,
> Kunal.

--
Harsh J

+1

On Fri, Mar 2, 2012 at 4:09 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Since you ask about anything in general, when I forayed into using
> Hadoop, my biggest pain was lack of documentation clarity and
> completeness over the MR and DFS user APIs (and other little points).
>
> It would be nice to have some work done to have one example or
> semi-example for every single Input/OutputFormat, Mapper/Reducer
> implementations, etc. added to the javadocs.
>
> I believe examples and snippets help out a ton (tons more than
> explaining just behavior) to new devs.
>
> On Fri, Mar 2, 2012 at 9:45 PM, Kunaal <[EMAIL PROTECTED]> wrote:
> > I am doing a general poll on what are the most prevalent pain points that
> > people run into with Hadoop? These could be performance related (memory
> > usage, IO latencies), usage related or anything really.
> >
> > The goal is to look for what areas this platform could benefit the most
> in
> > the near future.
> >
> > Any feedback is much appreciated.
> >
> > Thanks,
> > Kunal.
>
>
>
> --
> Harsh J
>

+2

Russell Jurney http://datasyndrome.com

On Mar 2, 2012, at 4:38 PM, Mohit Anchlia <[EMAIL PROTECTED]> wrote:

> +1
>
> On Fri, Mar 2, 2012 at 4:09 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> Since you ask about anything in general, when I forayed into using
>> Hadoop, my biggest pain was lack of documentation clarity and
>> completeness over the MR and DFS user APIs (and other little points).
>>
>> It would be nice to have some work done to have one example or
>> semi-example for every single Input/OutputFormat, Mapper/Reducer
>> implementations, etc. added to the javadocs.
>>
>> I believe examples and snippets help out a ton (tons more than
>> explaining just behavior) to new devs.
>>
>> On Fri, Mar 2, 2012 at 9:45 PM, Kunaal <[EMAIL PROTECTED]> wrote:
>>> I am doing a general poll on what are the most prevalent pain points that
>>> people run into with Hadoop? These could be performance related (memory
>>> usage, IO latencies), usage related or anything really.
>>>
>>> The goal is to look for what areas this platform could benefit the most
>> in
>>> the near future.
>>>
>>> Any feedback is much appreciated.
>>>
>>> Thanks,
>>> Kunal.
>>
>>
>>
>> --
>> Harsh J
>>

+3.14159265358979

Sent from my phone

On Mar 2, 2012, at 6:42 PM, Russell Jurney <[EMAIL PROTECTED]> wrote:

> +2
>
> Russell Jurney http://datasyndrome.com
>
> On Mar 2, 2012, at 4:38 PM, Mohit Anchlia <[EMAIL PROTECTED]> wrote:
>
>> +1
>>
>> On Fri, Mar 2, 2012 at 4:09 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>>> Since you ask about anything in general, when I forayed into using
>>> Hadoop, my biggest pain was lack of documentation clarity and
>>> completeness over the MR and DFS user APIs (and other little points).
>>>
>>> It would be nice to have some work done to have one example or
>>> semi-example for every single Input/OutputFormat, Mapper/Reducer
>>> implementations, etc. added to the javadocs.
>>>
>>> I believe examples and snippets help out a ton (tons more than
>>> explaining just behavior) to new devs.
>>>
>>> On Fri, Mar 2, 2012 at 9:45 PM, Kunaal <[EMAIL PROTECTED]> wrote:
>>>> I am doing a general poll on what are the most prevalent pain points that
>>>> people run into with Hadoop? These could be performance related (memory
>>>> usage, IO latencies), usage related or anything really.
>>>>
>>>> The goal is to look for what areas this platform could benefit the most
>>> in
>>>> the near future.
>>>>
>>>> Any feedback is much appreciated.
>>>>
>>>> Thanks,
>>>> Kunal.
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>

+6.28318531

On Fri, Mar 2, 2012 at 7:35 PM, Leonardo Urbina <[EMAIL PROTECTED]> wrote:

> +3.14159265358979
>
> Sent from my phone
>
> On Mar 2, 2012, at 6:42 PM, Russell Jurney <[EMAIL PROTECTED]>
> wrote:
>
> > +2
> >
> > Russell Jurney http://datasyndrome.com
> >
> > On Mar 2, 2012, at 4:38 PM, Mohit Anchlia <[EMAIL PROTECTED]>
> wrote:
> >
> >> +1
> >>
> >> On Fri, Mar 2, 2012 at 4:09 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> >>
> >>> Since you ask about anything in general, when I forayed into using
> >>> Hadoop, my biggest pain was lack of documentation clarity and
> >>> completeness over the MR and DFS user APIs (and other little points).
> >>>
> >>> It would be nice to have some work done to have one example or
> >>> semi-example for every single Input/OutputFormat, Mapper/Reducer
> >>> implementations, etc. added to the javadocs.
> >>>
> >>> I believe examples and snippets help out a ton (tons more than
> >>> explaining just behavior) to new devs.
> >>>
> >>> On Fri, Mar 2, 2012 at 9:45 PM, Kunaal <[EMAIL PROTECTED]>
> wrote:
> >>>> I am doing a general poll on what are the most prevalent pain points
> that
> >>>> people run into with Hadoop? These could be performance related
> (memory
> >>>> usage, IO latencies), usage related or anything really.
> >>>>
> >>>> The goal is to look for what areas this platform could benefit the
> most
> >>> in
> >>>> the near future.
> >>>>
> >>>> Any feedback is much appreciated.
> >>>>
> >>>> Thanks,
> >>>> Kunal.
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>>
>

--
Russell Jurney twitter.com/rjurney [EMAIL PROTECTED] datasyndrome.com

I found a related question here:

http://www.quora.com/Hadoop-Operability/Is-Hadoop-now-easy-to-use-If-not-what-assistance-do-most-users-need

Jie

On Fri, Mar 2, 2012 at 10:55 PM, Russell Jurney <[EMAIL PROTECTED]>wrote:

> +6.28318531
>
> On Fri, Mar 2, 2012 at 7:35 PM, Leonardo Urbina <[EMAIL PROTECTED]> wrote:
>
> > +3.14159265358979
> >
> > Sent from my phone
> >
> > On Mar 2, 2012, at 6:42 PM, Russell Jurney <[EMAIL PROTECTED]>
> > wrote:
> >
> > > +2
> > >
> > > Russell Jurney http://datasyndrome.com
> > >
> > > On Mar 2, 2012, at 4:38 PM, Mohit Anchlia <[EMAIL PROTECTED]>
> > wrote:
> > >
> > >> +1
> > >>
> > >> On Fri, Mar 2, 2012 at 4:09 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> > >>
> > >>> Since you ask about anything in general, when I forayed into using
> > >>> Hadoop, my biggest pain was lack of documentation clarity and
> > >>> completeness over the MR and DFS user APIs (and other little points).
> > >>>
> > >>> It would be nice to have some work done to have one example or
> > >>> semi-example for every single Input/OutputFormat, Mapper/Reducer
> > >>> implementations, etc. added to the javadocs.
> > >>>
> > >>> I believe examples and snippets help out a ton (tons more than
> > >>> explaining just behavior) to new devs.
> > >>>
> > >>> On Fri, Mar 2, 2012 at 9:45 PM, Kunaal <[EMAIL PROTECTED]>
> > wrote:
> > >>>> I am doing a general poll on what are the most prevalent pain points
> > that
> > >>>> people run into with Hadoop? These could be performance related
> > (memory
> > >>>> usage, IO latencies), usage related or anything really.
> > >>>>
> > >>>> The goal is to look for what areas this platform could benefit the
> > most
> > >>> in
> > >>>> the near future.
> > >>>>
> > >>>> Any feedback is much appreciated.
> > >>>>
> > >>>> Thanks,
> > >>>> Kunal.
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Harsh J
> > >>>
> >
>
>
>
> --
> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
> datasyndrome.com
>

2012/3/2 Kunaal wrote:
> I am doing a general poll on what are the most prevalent pain points that
> people run into with Hadoop? These could be performance related (memory
> usage, IO latencies), usage related or anything really.
>

My biggest frustration with core Hadoop after the last year or so has
been not having the capability to efficiently implement the so-called
"analytic functions" in general with map reduce.

These are not what one would think they are from just the name by the
way - see Oracle Analytics as an example of what I mean. The big
advantage is that they often allow you to avoid expensive self-joins
which can make a huge difference performance wise.

(I would say that 80% of the analytic functions can be implemented with
a UDF or a UDA in hive -- things like lead() or lag() or first() or
rank() -- but it is the other 20% that would knock the ball out of the park)

On Mar 2, 2012, at 4:09 PM, Harsh J wrote:

> Since you ask about anything in general, when I forayed into using
> Hadoop, my biggest pain was lack of documentation clarity and
> completeness over the MR and DFS user APIs (and other little points).
>
> It would be nice to have some work done to have one example or
> semi-example for every single Input/OutputFormat, Mapper/Reducer
> implementations, etc. added to the javadocs.
>
> I believe examples and snippets help out a ton (tons more than
> explaining just behavior) to new devs.

Good points Harsh. Would you like to contribute some documentation patches?
>
> On Fri, Mar 2, 2012 at 9:45 PM, Kunaal <[EMAIL PROTECTED]> wrote:
>> I am doing a general poll on what are the most prevalent pain points that
>> people run into with Hadoop? These could be performance related (memory
>> usage, IO latencies), usage related or anything really.
>>
>> The goal is to look for what areas this platform could benefit the most in
>> the near future.
>>
>> Any feedback is much appreciated.
>>
>> Thanks,
>> Kunal.
>
>
>
> --
> Harsh J

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

What?
The lack of documentation is what made Hadoop, really HBase, a lot of fun:-)
You know what they say... Not guts, no glory...

I'm sorry, while I agree w Harsh, I just don't want to sound like some old guy talking about how when they were young, they had to walk in chest high snow, in a blizzard, uphill (both ways)to and from school ... And how you newbies have it so much better...

;-P

Sent from my iPhone

On Mar 2, 2012, at 6:42 PM, "Russell Jurney" <[EMAIL PROTECTED]> wrote:

> +2
>
> Russell Jurney http://datasyndrome.com
>
> On Mar 2, 2012, at 4:38 PM, Mohit Anchlia <[EMAIL PROTECTED]> wrote:
>
>> +1
>>
>> On Fri, Mar 2, 2012 at 4:09 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>>> Since you ask about anything in general, when I forayed into using
>>> Hadoop, my biggest pain was lack of documentation clarity and
>>> completeness over the MR and DFS user APIs (and other little points).
>>>
>>> It would be nice to have some work done to have one example or
>>> semi-example for every single Input/OutputFormat, Mapper/Reducer
>>> implementations, etc. added to the javadocs.
>>>
>>> I believe examples and snippets help out a ton (tons more than
>>> explaining just behavior) to new devs.
>>>
>>> On Fri, Mar 2, 2012 at 9:45 PM, Kunaal <[EMAIL PROTECTED]> wrote:
>>>> I am doing a general poll on what are the most prevalent pain points that
>>>> people run into with Hadoop? These could be performance related (memory
>>>> usage, IO latencies), usage related or anything really.
>>>>
>>>> The goal is to look for what areas this platform could benefit the most
>>> in
>>>> the near future.
>>>>
>>>> Any feedback is much appreciated.
>>>>
>>>> Thanks,
>>>> Kunal.
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>