Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Hadoop overhead


Copy link to this message
-
Re: Hadoop overhead
If its too many short duration jobs, you might want to keep an eye on
jobtracker and tweak number of heartbeats processed per second &
outofbandheartbeat option. JobTracker might be bombarded with events
otherwise.

On Thu, Apr 8, 2010 at 8:07 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:

> By default, for each task hadoop will create a new jvm process which will
> be
> the major cost in my opinion. You can customize configuration to let
> tasktracker reuse the jvm to eliminate the overhead to some extend.
>
> On Thu, Apr 8, 2010 at 8:55 PM, Aleksandar Stupar <
> [EMAIL PROTECTED]> wrote:
>
> > Hi all,
> >
> > As I realize hadoop is mainly used for tasks that take long
> > time to execute. I'm considering to use hadoop for task
> > whose lower bound in distributed execution is like 5 to 10
> > seconds. Am wondering what would the overhead be with
> > using hadoop.
> >
> > Does anyone have an idea? Any link where I can find this out?
> >
> > Thanks,
> > Aleksandar.
> >
> >
> >
>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

--
~Rajesh.B