Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # dev - Questions about recommendation value of the "io.sort.mb" parameter


Copy link to this message
-
Re: Questions about recommendation value of the "io.sort.mb" parameter
Yu Li 2010-06-24, 00:42
Hi Todd,

Thanks a lot for your further explanation! It makes me more clear about this
parameter.

BTW, please allow me to express my thankfulness to everyone helps.

Best Regards,
Carp

在 2010年6月24日 上午1:49,Todd Lipcon <[EMAIL PROTECTED]>写道:

> Plus there's some overhead for each record of map output. Specifically, 24
> bytes. So if you output 64MB worth of data, but each of your objects is
> only
> 24 bytes long itself, you need more than 128MB worth of spill space for it.
> Last, the map output buffer begins spilling when it is partially full so
> that more records can be collected while spill proceeds.
>
> 200MB io.sort.mb has enough headroom for most 64M input splits that don't
> blow up the data a lot. Expanding much above 200M for most jobs doesn't buy
> you much. Good news is it's easy to tell by looking at the logs to see how
> many times the map tasks are spilling. If you're only spilling once, more
> io.sort.mb will not help.
>
> -Todd
>
> 2010/6/23 李钰 <[EMAIL PROTECTED]>
>
> > Hi Jeff,
> >
> > Thanks for your quick reply. Seems my thinking is stuck on the job style
> > I'm
> > running. Now I'm much clearer about it.
> >
> > Best Regards,
> > Carp
> >
> > 2010/6/23 Jeff Zhang <[EMAIL PROTECTED]>
> >
> > > Hi 李钰
> > >
> > > The size of map output depends on your Mapper class. The Mapper class
> > > will do processing on the input data.
> > >
> > >
> > >
> > > 2010/6/23 李钰 <[EMAIL PROTECTED]>:
> > >  > Hi Sriguru,
> > > >
> > > > Thanks a lot for your comments and suggestions!
> > > > Here I still have some questions: since map mainly do data
> preparation,
> > > > say split input data into KVPs, sort and partition before spill,
> would
> > > the
> > > > size of map output KVPs be much larger than the input data size? If
> > not,
> > > > since one map task deals with one input split, and one input split is
> > > > usually 64M, the map KVPs size would be proximately 64M. Could you
> > please
> > > > give me some example on map output much larger than the input split?
> It
> > > > really confuse me for some time, thanks.
> > > >
> > > > Others,
> > > >
> > > > Also badly need your help if you know about this, thanks.
> > > >
> > > > Best Regards,
> > > > Carp
> > > >
> > > > 在 2010年6月23日 下午5:11,Srigurunath Chakravarthi <[EMAIL PROTECTED]
> > >写道:
> > > >
> > > >> Hi Carp,
> > > >>  Your assumption is right that this is a per-map-task setting.
> > > >> However, this buffer stores map output KVPs, not input. Therefore
> the
> > > >> optimal value depends on how much data your map task is generating.
> > > >>
> > > >> If your output per map is greater than io.sort.mb, these rules of
> > thumb
> > > >> that could work for you:
> > > >>
> > > >> 1) Increase max heap of map tasks to use RAM better, but not hit
> swap.
> > > >> 2) Set io.sort.mb to ~70% of heap.
> > > >>
> > > >> Overall, causing extra "spills" (because of insufficient io.sort.mb)
> > is
> > > >> much better than risking swapping (by setting io.sort.mb and heap
> too
> > > >> large), in terms of relative performance penalty you will pay.
> > > >>
> > > >> Cheers,
> > > >> Sriguru
> > > >>
> > > >> >-----Original Message-----
> > > >> >From: 李钰 [mailto:[EMAIL PROTECTED]]
> > > >> >Sent: Wednesday, June 23, 2010 12:27 PM
> > > >> >To: [EMAIL PROTECTED]
> > > >> >Subject: Questions about recommendation value of the "io.sort.mb"
> > > >> >parameter
> > > >> >
> > > >> >Dear all,
> > > >> >
> > > >> >Here I've got a question about the "io.sort.mb" parameter. We can
> > find
> > > >> >material from Yahoo! or Cloudera which recommend setting this value
> > to
> > > >> >200
> > > >> >if the job scale is large, but I'm confused about this. As I know,
> > > >> >the tasktracker will launch a child-JVM for each task, and
> > > >> >“*io.sort.mb*”
> > > >> >presents the buffer size in memory inside *one map task child-JVM*,
> > the
> > > >> >default value 100MB should be large enough because the input split
> of
> > > >> >one
> > > >> >map task is usually 64MB, as large as the block size we usually