-Re: Hadoop overhead
Rajesh Balamohan 2010-04-08, 14:50
If its too many short duration jobs, you might want to keep an eye on
jobtracker and tweak number of heartbeats processed per second &
outofbandheartbeat option. JobTracker might be bombarded with events
On Thu, Apr 8, 2010 at 8:07 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> By default, for each task hadoop will create a new jvm process which will
> the major cost in my opinion. You can customize configuration to let
> tasktracker reuse the jvm to eliminate the overhead to some extend.
> On Thu, Apr 8, 2010 at 8:55 PM, Aleksandar Stupar <
> [EMAIL PROTECTED]> wrote:
> > Hi all,
> > As I realize hadoop is mainly used for tasks that take long
> > time to execute. I'm considering to use hadoop for task
> > whose lower bound in distributed execution is like 5 to 10
> > seconds. Am wondering what would the overhead be with
> > using hadoop.
> > Does anyone have an idea? Any link where I can find this out?
> > Thanks,
> > Aleksandar.
> Best Regards
> Jeff Zhang