Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - performance of a hbase map/reduce job
Ey-Chih chow
2012-05-30, 01:52
Ted Yu
2012-05-30, 02:39
Ey-Chih chow
2012-05-30, 06:48
Ey-Chih chow
2012-05-30, 07:15
Copy link to this message
-
Re: performance of a hbase map/reduce job
Kevin O'dell 2012-05-31, 14:13
It only matters if you are using them all.

On Wed, May 30, 2012 at 12:15 AM, Ey-Chih chow <[EMAIL PROTECTED]> wrote:

> By the way, the number of maximal client connections is set to default
> value, i.e. 60.  Does this matter?  Thanks.
>
> Ey-Chih Chow
>
>
> On May 29, 2012, at 11:48 PM, Ey-Chih chow wrote:
>
> > It's a 3 node cluster running zookeeper-3.3.3 no modifications
> > have been made by us. This is separate and dedicated only for hbase.
> >
> > We were looking at the ganglia graphs, not much load almost none,
> > traffic is 1-2kb/sec.
> >
> > Thanks.
> >
> > Ey-Chih Chow
> >
> >
> > On May 29, 2012, at 7:39 PM, Ted Yu wrote:
> >
> >> Can you describe your zookeeper setup a little more ?
> >>
> >> Thanks
> >>
> >> On Tue, May 29, 2012 at 6:52 PM, Ey-Chih chow <[EMAIL PROTECTED]> wrote:
> >>
> >>> Hi,
> >>>
> >>> We have some performance issue with a hbase map-only job, that copies
> data
> >>> from hbase to hdfs.  We profiled the job and what follows were the top
> 6
> >>> ranked methods, in terms of CPU usages, and the corresponding traces.
> >>>
> >>>
> =========================================================================> >>>
> >>> CPU SAMPLES BEGIN (total = 1162350) Tue May 29 16:54:18 2012
> >>> rank   self  accum   count trace method
> >>> 1 16.67% 16.67%  193724 312698 sun.nio.ch.EPollArrayWrapper.epollWait
> >>> 2 16.66% 33.32%  193609 314324 sun.nio.ch.EPollArrayWrapper.epollWait
> >>> 3 16.65% 49.98%  193573 314044 sun.nio.ch.EPollArrayWrapper.epollWait
> >>> 4 16.56% 66.54%  192491 319621 sun.nio.ch.EPollArrayWrapper.epollWait
> >>> 5 10.90% 77.44%  126699 319206 sun.nio.ch.EPollArrayWrapper.epollWait
> >>> 6  5.64% 83.08%   65565 327226 sun.nio.ch.EPollArrayWrapper.epollWait
> >>>
> >>> TRACE 312698: (thread=200010)
> >>>
> >>> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java:Unknown
> line)
> >>>      sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> >>>      sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> >>>      sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> >>>      sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> >>>
>  org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> >>>
> >>> TRACE 314324: (thread=200015)
> >>>
> >>> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java:Unknown
> line)
> >>>      sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> >>>      sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> >>>      sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> >>>      sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> >>>
>  org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> >>>
> >>> TRACE 314044: (thread=200013)
> >>>
> >>> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java:Unknown
> line)
> >>>      sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> >>>      sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> >>>      sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> >>>      sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> >>>
>  org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> >>>
> >>> TRACE 319621: (thread=200023)
> >>>
> >>> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java:Unknown
> line)
> >>>      sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> >>>      sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> >>>      sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> >>>      sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> >>>
>  org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> >>>
> >>> TRACE 319206: (thread=200022)
> >>>
> >>> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java:Unknown
> line)
> >>>      sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> >>>      sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> >>>      sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
Kevin O'Dell
Customer Operations Engineer, Cloudera