Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> RegionServer shutdown with ScanWildcardColumnTracker exception


+
Amit Sela 2013-04-17, 08:30
+
ramkrishna vasudevan 2013-04-17, 08:47
+
Amit Sela 2013-04-17, 09:06
Copy link to this message
-
Re: RegionServer shutdown with ScanWildcardColumnTracker exception
Hi Amit

Checking the code this is possible when the qualifiers are not sorted.  Do
you have any CPs in your path which tries to play with the KVs?

Seems to be a very weird thing.
Can you try doing a scan on the KV just before this happens.  That will tel
you the existing kvs that are present.

Even now if you can have the cluster you can try scanning for the region
for which the flush happened.  That will give us some more info.

Regards
Ram
On Wed, Apr 17, 2013 at 2:36 PM, Amit Sela <[EMAIL PROTECTED]> wrote:

> The cluster runs Hadoop 1.0.4 and HBase 0.94.2
>
> I have three families in this table: weekly, daily, hourly. each family has
> the following qualifiers:
> Weekly - impressions_{countrycode}_{week#} - country code is 0, 1 or ALL
> (aggregation of both 0 and 1)
> Daily and hourly are the same but with yyyyMMdd and yyyyMMddhh
> respectively.
>
> Just before the exception the regionserver StoreFile executes the
> following:
>
> 2013-04-16 17:56:06,769 [regionserver8041.cacheFlusher] INFO
> org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter
> type for hdfs://
> hadoop-master.infolinks.com:8000/hbase/URL_COUNTERS/af2760e
> 4d04a9e3025d1fb53bdba8acf/.tmp/dc4ce516887f4e0bbaf6201d69ba90bc:
> CompoundBloomFilterWriter
> 2013-04-16 17:56:07,331 [regionserver8041.cacheFlusher] INFO
> org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO
> DeleteFamily was added to HFile (hdfs://hbase-master-address:8000/hbase
> /URL_COUNTERS/*af2760e4d04a9e3025d1fb53bdba8acf*
> /.tmp/dc4ce516887f4e0bbaf6201d69ba90bc)
> 2013-04-16 17:56:07,331 [regionserver8041.cacheFlusher] INFO
> org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=210517246,
> memsize=39.3m, into tmp file hdfs://hbase-master:8000/hbase
> /URL_COUNTERS/*af2760e4d04a9e3025d1fb53bdba8acf*
> /.tmp/dc4ce516887f4e0bbaf6201d69ba90bc
> 2013-04-16 17:56:07,357 [regionserver8041.cacheFlusher] INFO
> org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter
> type for hdfs://hbase-master:8000/hbase/URL_COUNTERS/*af2760e*
> *4d04a9e3025d1fb53bdba8acf*/.tmp/3fa7993dcb294be1bca5e4d7357f4003:
> CompoundBloomFilterWriter
> 2013-04-16 17:56:07,608 [regionserver8041.cacheFlusher] INFO
> org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO
> DeleteFamily was added to HFile (hdfs://hbase-master:8000/hbase
> /URL_COUNTERS/*af2760e4d04a9e3025d1fb53bdba8acf*
> /.tmp/3fa7993dcb294be1bca5e4d7357f4003)
> 2013-04-16 17:56:07,608 [regionserver8041.cacheFlusher] FATAL
> org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
> region-server-address,8041,1364993168088: Replay of HLog required
> . Forcing server shutdown
> DroppedSnapshotException: region: TABLE,ROWKEY,1364317591568.*
> af2760e4d04a9e3025d1fb53bdba8acf*.
> ....
> ....
> ...
>
>
> On Wed, Apr 17, 2013 at 11:47 AM, ramkrishna vasudevan <
> [EMAIL PROTECTED]> wrote:
>
> > Seems interesting.  Can  you tell us what are the families and the
> > qualifiers available in your schema.
> >
> > Any other interesting logs that you can see before this?
> >
> > BTW the version of HBase is also needed?  If we can track it out we can
> > then file a JIRA if it is a bug.
> >
> > Regards
> > RAm
> >
> >
> > On Wed, Apr 17, 2013 at 2:00 PM, Amit Sela <[EMAIL PROTECTED]> wrote:
> >
> > > Hi all,
> > >
> > > I had a regionserver crushed during counters increment. Looking at the
> > > regionserver log I saw:
> > >
> > > org.apache.hadoop.hbase.DroppedSnapshotException: region: TABLE_NAME,
> > > ROW_KEY...at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1472)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1351)
> > >         at
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1292)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:406)
+
Amit Sela 2013-04-17, 11:47
+
ramkrishna vasudevan 2013-04-17, 15:56
+
Amit Sela 2013-04-17, 16:10
+
ramkrishna vasudevan 2013-04-17, 16:17
+
ramkrishna vasudevan 2013-04-17, 17:18
+
Amit Sela 2013-04-17, 18:34
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB