Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Re: Hbase import Tsv performance (slow import)


+
Nick maillard 2012-10-24, 11:40
+
ramkrishna vasudevan 2012-10-24, 13:47
+
Nick maillard 2012-10-24, 10:15
+
Sonal Goyal 2012-10-24, 11:18
+
Nick maillard 2012-10-24, 10:05
+
Nick maillard 2012-10-24, 09:23
+
Nick maillard 2012-10-24, 14:35
+
Kevin Odell 2012-10-24, 16:18
Copy link to this message
-
Re: Hbase import Tsv performance (slow import)
anil gupta 2012-10-24, 16:30
Hi Nick,

How many hard drives your slaves has? RPM of those? How many mappers are
run concurrently on a node?Did you turn off speculative execution? Have a
look at disk i/o to see whether that is a bottleneck or not.

MR is disk I/O bound so if you only have one disk on slave and you are
running 5 Mapper concurrently then the job will slow down.

Thanks,
Anil

On Wed, Oct 24, 2012 at 9:18 AM, Kevin O'dell <[EMAIL PROTECTED]>wrote:

> Nick,
>
>   What versions are you using:
>
> HDFS
> HBase
> OS
>  On Oct 24, 2012 10:36 AM, "Nick maillard" <
> [EMAIL PROTECTED]>
> wrote:
>
> > Hello everyone
> >
> > Still looking in the issue.
> > I have tried different tests and the results are surprising.
> > If I put mapred.tasktracker.map.tasks.maximum: 28
> > I get a total of 84 tasks on my cluster and the process takes about 1h15
> > min
> > each task taking up 1h10 minutes. The whole file being cut down in 80
> > tasks.
> >
> > If I put  mapred.tasktracker.map.tasks.maximum: 3
> > I get a total of 6 tasks on my cluster and the process takes about the
> same
> > amount of time 1h20 still cutting down the whole file in 80 tasks, but
> now
> > of
> > course each individual task only takes up a couple of minutes.
> >
> > It's like the overall importTSv must take 1h something and the duration
> of
> > the
> > map tasks vary accordingly.
> >
> > There is definitly something I am doing wrong.
> >
> >
> >
> >
>

--
Thanks & Regards,
Anil Gupta
+
Nick maillard 2012-10-24, 16:29
+
nick maillard 2012-10-24, 19:08
+
Nick maillard 2012-10-23, 17:13
+
Nicolas Liochon 2012-10-23, 17:32
+
Kevin Odell 2012-10-23, 17:47
+
lars hofhansl 2012-10-25, 04:10
+
Nick maillard 2012-10-23, 15:48
+
Anoop John 2012-10-24, 03:29
+
ramkrishna vasudevan 2012-10-24, 04:55
+
anil gupta 2012-10-24, 05:09
+
Anoop John 2012-10-24, 05:11
+
Anoop John 2012-10-24, 05:14
+
anil gupta 2012-10-24, 05:28
+
Anoop John 2012-10-24, 06:07
+
anil gupta 2012-10-24, 06:14
+
Anoop John 2012-10-24, 06:31
+
anil gupta 2012-10-24, 06:43
+
ramkrishna vasudevan 2012-10-24, 05:52
+
anil gupta 2012-10-24, 06:11
+
Jonathan Bishop 2012-10-25, 15:57
+
anil gupta 2012-10-25, 20:33
+
anil gupta 2012-10-25, 20:35
+
Anoop Sam John 2012-10-26, 04:07
+
Nicolas Liochon 2012-10-23, 16:46