Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - RegionServer shutdown with ScanWildcardColumnTracker exception


Copy link to this message
-
Re: RegionServer shutdown with ScanWildcardColumnTracker exception
Amit Sela 2013-04-17, 09:06
The cluster runs Hadoop 1.0.4 and HBase 0.94.2

I have three families in this table: weekly, daily, hourly. each family has
the following qualifiers:
Weekly - impressions_{countrycode}_{week#} - country code is 0, 1 or ALL
(aggregation of both 0 and 1)
Daily and hourly are the same but with yyyyMMdd and yyyyMMddhh
respectively.

Just before the exception the regionserver StoreFile executes the
following:

2013-04-16 17:56:06,769 [regionserver8041.cacheFlusher] INFO
org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter
type for hdfs://hadoop-master.infolinks.com:8000/hbase/URL_COUNTERS/af2760e
4d04a9e3025d1fb53bdba8acf/.tmp/dc4ce516887f4e0bbaf6201d69ba90bc:
CompoundBloomFilterWriter
2013-04-16 17:56:07,331 [regionserver8041.cacheFlusher] INFO
org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO
DeleteFamily was added to HFile (hdfs://hbase-master-address:8000/hbase
/URL_COUNTERS/*af2760e4d04a9e3025d1fb53bdba8acf*
/.tmp/dc4ce516887f4e0bbaf6201d69ba90bc)
2013-04-16 17:56:07,331 [regionserver8041.cacheFlusher] INFO
org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=210517246,
memsize=39.3m, into tmp file hdfs://hbase-master:8000/hbase
/URL_COUNTERS/*af2760e4d04a9e3025d1fb53bdba8acf*
/.tmp/dc4ce516887f4e0bbaf6201d69ba90bc
2013-04-16 17:56:07,357 [regionserver8041.cacheFlusher] INFO
org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter
type for hdfs://hbase-master:8000/hbase/URL_COUNTERS/*af2760e*
*4d04a9e3025d1fb53bdba8acf*/.tmp/3fa7993dcb294be1bca5e4d7357f4003:
CompoundBloomFilterWriter
2013-04-16 17:56:07,608 [regionserver8041.cacheFlusher] INFO
org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO
DeleteFamily was added to HFile (hdfs://hbase-master:8000/hbase
/URL_COUNTERS/*af2760e4d04a9e3025d1fb53bdba8acf*
/.tmp/3fa7993dcb294be1bca5e4d7357f4003)
2013-04-16 17:56:07,608 [regionserver8041.cacheFlusher] FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
region-server-address,8041,1364993168088: Replay of HLog required
. Forcing server shutdown
DroppedSnapshotException: region: TABLE,ROWKEY,1364317591568.*
af2760e4d04a9e3025d1fb53bdba8acf*.
....
....
...
On Wed, Apr 17, 2013 at 11:47 AM, ramkrishna vasudevan <
[EMAIL PROTECTED]> wrote:

> Seems interesting.  Can  you tell us what are the families and the
> qualifiers available in your schema.
>
> Any other interesting logs that you can see before this?
>
> BTW the version of HBase is also needed?  If we can track it out we can
> then file a JIRA if it is a bug.
>
> Regards
> RAm
>
>
> On Wed, Apr 17, 2013 at 2:00 PM, Amit Sela <[EMAIL PROTECTED]> wrote:
>
> > Hi all,
> >
> > I had a regionserver crushed during counters increment. Looking at the
> > regionserver log I saw:
> >
> > org.apache.hadoop.hbase.DroppedSnapshotException: region: TABLE_NAME,
> > ROW_KEY...at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1472)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1351)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1292)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:406)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:380)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:243)
> >         at java.lang.Thread.run(Thread.java:722)
> > Caused by: java.io.IOException: ScanWildcardColumnTracker.checkColumn ran
> > into a column actually smaller than the previous column: *QUALIFIER*
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.ScanWildcardColumnTracker.checkColumn(ScanWildcardColumnTracker.java:104)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:354)
> >         at
> >