Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Bulk loading job failed when one region server went down in the cluster


Copy link to this message
-
Re: Bulk loading job failed when one region server went down in the cluster
Hi Mike,

Here is the link to my email on Hadoop list regarding YARN problem:
http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201208.mbox/%3CCAF1+Vs8oF4VsHbg14B7SGzBB_8Ty7GC9Lw3nm1bM0v+[EMAIL PROTECTED]%3E

Somehow the link for cloudera mail in last email does not seems to work.
Here is the new link:
https://groups.google.com/a/cloudera.org/forum/?fromgroups#!searchin/cdh-user/yarn$20anil/cdh-user/J564g9A8tPE/ZpslzOkIGZYJ%5B1-25%5D

Thanks for your help,
Anil Gupta

On Mon, Aug 13, 2012 at 1:14 PM, anil gupta <[EMAIL PROTECTED]> wrote:

> Hi Mike,
>
> I tried doing that by setting up properties in mapred-site.xml but Yarn
> doesnt seems to work with "mapreduce.tasktracker.
> map.tasks.maximum" property. Here is a reference to a discussion to same
> problem:
>
> https://groups.google.com/a/cloudera.org/forum/?fromgroups#!searchin/cdh-user/yarn$20anil/cdh-user/J564g9A8tPE/ZpslzOkIGZYJ[1-25]
> I have also posted about the same problem in Hadoop mailing list.
>
> I already admitted in my previous email that YARN is having major issues
> when we want to control it in low memory environment. I was just trying to
> get views HBase experts on bulk load failures since we will be relying
> heavily on Fault Tolerance.
> If HBase Bulk Loader is fault tolerant to failure of RS in a viable
> environment  then I dont have any issue. I hope this clears up my purpose
> of posting on this topic.
>
> Thanks,
> Anil
>
> On Mon, Aug 13, 2012 at 12:39 PM, Michael Segel <[EMAIL PROTECTED]
> > wrote:
>
>> Anil,
>>
>> Do you know what happens when you have an airplane that has too heavy a
>> cargo when it tries to take off?
>> You run out of runway and you crash and burn.
>>
>> Looking at your post, why are you starting 8 map processes on each slave?
>> That's tunable and you clearly do not have enough memory in each VM to
>> support 8 slots on a node.
>> Here you swap, you swap you cause HBase to crash and burn.
>>
>> 3.2GB of memory means that no more than 1 slot per slave and even then...
>> you're going to be very tight. Not to mention that you will need to loosen
>> up on your timings since its all virtual and you have way too much i/o per
>> drive going on.
>>
>>
>> My suggestion is that you go back and tune your system before thinking
>> about running anything.
>>
>> HTH
>>
>> -Mike
>>
>> On Aug 13, 2012, at 2:11 PM, anil gupta <[EMAIL PROTECTED]> wrote:
>>
>> > Hi Guys,
>> >
>> > Sorry for not mentioning the version I am currently running. My current
>> > version is HBase 0.92.1(cdh4) and running Hadoop2.0.0-Alpha with YARN
>> for
>> > MR. My original post was for HBase0.92. Here are some more details of my
>> > current setup:
>> > I am running a 8 slave, 4 admin node cluster on CentOS6.0 VM's
>> installed on
>> > VMware Hyprevisor 5.0. Each of my VM is having 3.2 GB of memory and 500
>> > HDFS space.
>> > I use this cluster for POC(Proof of Concepts). I am not looking for any
>> > performance benchmarking from this set-up. Due to some major bugs in
>> YARN i
>> > am unable to make work in a proper way in memory less than 4GB. I am
>> > already having discussion regarding them on Hadoop Mailing List.
>> >
>> > Here is the log of failed mapper: http://pastebin.com/f83xE2wv
>> >
>> > The problem is that when i start a Bulk loading job in YARN, 8 Map
>> > processes start on each slave and then all of my slaves are hammered
>> badly
>> > due to this. Since the slaves are getting hammered badly then
>> RegionServer
>> > gets lease expired or YourAreDeadExpcetion. Here is the log of RS which
>> > caused the job to fail: http://pastebin.com/9ZQx0DtD
>> >
>> > I am aware that this is happening due to underperforming hardware(Two
>> > slaves are using one 7200 rpm Hard Drive in my setup) and some major
>> bugs
>> > regarding running YARN in less than 4 GB memory. My only concern is the
>> > failure of entire MR job and its fault tolerance to RS failures. I am
>> not
>> > really concerned about RS failure since HBase is fault tolerant.

Thanks & Regards,
Anil Gupta
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB