Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> region server down when scanning using mapreduce


+
Lu, Wei 2013-03-12, 05:06
+
Azuryy Yu 2013-03-12, 05:18
+
Lu, Wei 2013-03-12, 05:31
+
Azuryy Yu 2013-03-12, 05:41
+
Lu, Wei 2013-03-12, 08:12
Copy link to this message
-
RE: region server down when scanning using mapreduce
How is the GC pattern in your RSs which are getting down? In RS logs you might be having YouAreDeadExceptions...
Pls try tuning your RS memory and GC opts.

-Anoop-
________________________________________
From: Lu, Wei [[EMAIL PROTECTED]]
Sent: Tuesday, March 12, 2013 1:42 PM
To: [EMAIL PROTECTED]
Subject: RE: region server down when scanning using mapreduce

We turned the block cache to false and tried again, regionserver still crash one after another.
There are a lot of scanner lease time out, and then master log info:
        RegionServer ephemeral node deleted, processing expiration [rs21,60020,1363010589837]
Seems the problem is not caused by block cache
Thanks

-----Original Message-----
From: Azuryy Yu [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, March 12, 2013 1:41 PM
To: [EMAIL PROTECTED]
Subject: Re: region server down when scanning using mapreduce

please read here http://hbase.apache.org/book.html (11.8.5. Block Cache) to
get some background of block cache.
On Tue, Mar 12, 2013 at 1:31 PM, Lu, Wei <[EMAIL PROTECTED]> wrote:

> No, does block cache matter? Btw, the mr dump is a mr program we
> implemented rather than the hbase tool.
>
> Thanks
>
> -----Original Message-----
> From: Azuryy Yu [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 12, 2013 1:18 PM
> To: [EMAIL PROTECTED]
> Subject: Re: region server down when scanning using mapreduce
>
> did you closed block cache when you used mr dump?
> On Mar 12, 2013 1:06 PM, "Lu, Wei" <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > When we use mapreduce to dump data from a pretty large table on hbase.
> One
> > region server crash and then another. Mapreduce is deployed together with
> > hbase.
> >
> > 1) From log of the region server, there are both "next" and "multi"
> > operations on going. Is it because there is write/read conflict that
> cause
> > scanner timeout?
> > 2) Region server has 24 cores, and # max map tasks is 24 too; the table
> > has about 30 regions (each of size 0.5G) on the region server, is it
> > because cpu is all used by mapreduce and that case region server slow and
> > then timeout?
> > 2) current hbase.regionserver.handler.count is 10 by default, should it
> be
> > enlarged?
> >
> > Please give us some advices.
> >
> > Thanks,
> > Wei
> >
> >
> > Log information:
> >
> >
> > [Regionserver rs21:]
> >
> > 2013-03-11 18:36:28,148 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/
> > adcbg21.machine.wisdom.com
> ,60020,1363010589837/rs21%2C60020%2C1363010589837.1363025554488,
> > entries=22417, filesize=127539793.  for
> >
> /hbase/.logs/rs21,60020,1363010589837/rs21%2C60020%2C1363010589837.1363026988052
> > 2013-03-11 18:37:39,481 WARN org.apache.hadoop.hbase.util.Sleeper: We
> > slept 28183ms instead of 3000ms, this is likely due to a long garbage
> > collecting pause and it's usually bad, see
> > http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
> > 2013-03-11 18:37:40,163 WARN org.apache.hadoop.ipc.HBaseServer:
> > (responseTooSlow):
> > {"processingtimems":29830,"call":"next(1656517918313948447, 1000), rpc
> > version=1, client version=29, methodsFingerPrint=54742778","client":"
> > 10.20.127.21:56058
> >
> ","starttimems":1363027030280,"queuetimems":4602,"class":"HRegionServer","responsesize":2774484,"method":"next"}
> > 2013-03-11 18:37:40,163 WARN org.apache.hadoop.ipc.HBaseServer:
> > (responseTooSlow):
> > {"processingtimems":31195,"call":"next(-8353194140406556404, 1000), rpc
> > version=1, client version=29, methodsFingerPrint=54742778","client":"
> > 10.20.127.21:56529
> >
> ","starttimems":1363027028804,"queuetimems":3634,"class":"HRegionServer","responsesize":2270919,"method":"next"}
> > 2013-03-11 18:37:40,163 WARN org.apache.hadoop.ipc.HBaseServer:
> > (responseTooSlow):
> > {"processingtimems":30965,"call":"next(2623756537510669130, 1000), rpc
> > version=1, client version=29, methodsFingerPrint=54742778","client":"
> > 10.20.127.21:56146
> >
> ","starttimems":1363027028807,"queuetimems":3484,"class":"HRegionServer","responsesize":2753299,"method":"next"}
+
Azuryy Yu 2013-03-12, 13:13
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB