Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> hbase threw NotServingRegionException


+
Ey-Chih chow 2012-07-23, 19:39
+
Ey-Chih chow 2012-07-23, 19:56
+
Mohammad Tariq 2012-07-23, 20:10
+
Ey-Chih chow 2012-07-23, 21:05
Copy link to this message
-
Re: hbase threw NotServingRegionException
hbck should help expose more problems than a single scan would.  With that
said the logs are the best bet in understanding what is going on with the
cluster at the time of the issue.  You posted logs that seem to only really
contain GC info.  Do you have more information about the cluster state when
the exceptions were thrown ?  Do you have the logs from the master? Was the
RS responding at the time? Were there compactions or balancer actions
happening ?

On Mon, Jul 23, 2012 at 2:05 PM, Ey-Chih chow <[EMAIL PROTECTED]> wrote:

> Thanks.  But if we do a scan on the table via the hbase shell, data in the
> table did show up.
>
> Ey-Chih
>
> On Jul 23, 2012, at 1:10 PM, Mohammad Tariq wrote:
>
> > Hello sir,
> >
> >     A possible reason could be, your client is contacting the given
> > regionserver, and that regionserver kept on rejecting the requests.
> > Are you sure your table and all regions online? Use hbck once and see
> > if you find anything interesting.
> >
> > Regards,
> >    Mohammad Tariq
> >
> >
> > On Tue, Jul 24, 2012 at 1:26 AM, Ey-Chih chow <[EMAIL PROTECTED]> wrote:
> >> Sorry I pasted the wrong portion of the region server log.  The right
> portion should be as follows:
> >>
> >>
> >> =====================================> >> 2012-07-22T00:48:57.147-0700: [GC [ParNew: 18863K->2112K(19136K),
> 0.0029870 secs] 57106K->42831K(97048K) icms_dc=0 , 0.0030480 secs] [Times:
> user=0.01 sys=0.00, real=0.00 secs]
> >> 2012-07-22T00:49:09.475-0700: [GC [ParNew: 18686K->2112K(19136K),
> 0.0219960 secs] 59405K->43951K(97048K) icms_dc=0 , 0.0220530 secs] [Times:
> user=0.15 sys=0.00, real=0.02 secs]
> >> 2012-07-22T00:49:20.872-0700: [GC [ParNew: 19135K->2112K(19136K),
> 0.0091670 secs] 60974K->44659K(97048K) icms_dc=0 , 0.0092480 secs] [Times:
> user=0.06 sys=0.00, real=0.01 secs]
> >> 2012-07-22T00:49:22.285-0700: [GC [ParNew: 19136K->2112K(19136K),
> 0.0021870 secs] 61683K->46380K(97048K) icms_dc=0 , 0.0022480 secs] [Times:
> user=0.01 sys=0.00, real=0.00 secs]
> >> 2012-07-22T00:49:24.999-0700: [GC [ParNew: 19136K->2087K(19136K),
> 0.0019340 secs] 63404K->47842K(97048K) icms_dc=0 , 0.0019910 secs] [Times:
> user=0.01 sys=0.00, real=0.00 secs]
> >> 2012-07-22T00:49:26.618-0700: [GC [ParNew: 18002K->2112K(19136K),
> 0.0016160 secs] 63757K->48065K(97048K) icms_dc=0 , 0.0016780 secs] [Times:
> user=0.01 sys=0.00, real=0.00 secs]
> >> 2012-07-22T00:49:28.089-0700: [GC [ParNew: 19136K->1413K(19136K),
> 0.0033140 secs] 65089K->48748K(97048K) icms_dc=0 , 0.0033810 secs] [Times:
> user=0.01 sys=0.00, real=0.00 secs]
> >> 2012-07-22T00:49:29.352-0700: [GC [ParNew: 18423K->2112K(19136K),
> 0.0063630 secs] 65759K->50827K(97048K) icms_dc=0 , 0.0064210 secs] [Times:
> user=0.05 sys=0.00, real=0.01 secs]
> >> 2012-07-22T00:49:30.226-0700: [GC [ParNew: 19136K->1897K(19136K),
> 0.0021470 secs] 67851K->52058K(97048K) icms_dc=0 , 0.0022010 secs] [Times:
> user=0.01 sys=0.00, real=0.00 secs]
> >> 2012-07-22T00:49:31.190-0700: [GC [ParNew: 18921K->1606K(19136K),
> 0.0013270 secs] 69082K->52336K(97048K) icms_dc=0 , 0.0013780 secs] [Times:
> user=0.01 sys=0.00, real=0.00 secs]
> >> 2012-07-22T00:49:33.111-0700: [GC [ParNew: 18482K->2112K(19136K),
> 0.0017970 secs] 69212K->53496K(97048K) icms_dc=0 , 0.0018510 secs] [Times:
> user=0.01 sys=0.00, real=0.00 secs]
> >> 2012-07-22T00:49:34.396-0700: [GC [ParNew: 19136K->1468K(19136K),
> 0.0120290 secs] 70520K->54718K(97048K) icms_dc=0 , 0.0120820 secs] [Times:
> user=0.07 sys=0.00, real=0.02 secs]
> >> 2012-07-22T00:49:35.294-0700: [GC [ParNew: 18181K->2112K(19136K),
> 0.0011290 secs] 71431K->55399K(97048K) icms_dc=0 , 0.0011840 secs] [Times:
> user=0.01 sys=0.00, real=0.00 secs]
> >> 2012-07-22T00:49:39.165-0700: [GC [ParNew: 19003K->547K(19136K),
> 0.0130200 secs] 72291K->55364K(97048K) icms_dc=0 , 0.0130880 secs] [Times:
> user=0.08 sys=0.00, real=0.02 secs]
> >> 2012-07-22T00:49:40.642-0700: [GC [ParNew: 17571K->646K(19136K),
> 0.0006740 secs] 72388K->55463K(97048K) icms_dc=0 , 0.0007220 secs] [Times:
+
Ey-Chih chow 2012-07-23, 23:34
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB