Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> RegionServer shutdown with ScanWildcardColumnTracker exception


Copy link to this message
-
Re: RegionServer shutdown with ScanWildcardColumnTracker exception
The cluster runs Hadoop 1.0.4 and HBase 0.94.2

I have three families in this table: weekly, daily, hourly. each family has
the following qualifiers:
Weekly - impressions_{countrycode}_{week#} - country code is 0, 1 or ALL
(aggregation of both 0 and 1)
Daily and hourly are the same but with yyyyMMdd and yyyyMMddhh
respectively.

Just before the exception the regionserver StoreFile executes the
following:

2013-04-16 17:56:06,769 [regionserver8041.cacheFlusher] INFO
org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter
type for hdfs://hadoop-master.infolinks.com:8000/hbase/URL_COUNTERS/af2760e
4d04a9e3025d1fb53bdba8acf/.tmp/dc4ce516887f4e0bbaf6201d69ba90bc:
CompoundBloomFilterWriter
2013-04-16 17:56:07,331 [regionserver8041.cacheFlusher] INFO
org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO
DeleteFamily was added to HFile (hdfs://hbase-master-address:8000/hbase
/URL_COUNTERS/*af2760e4d04a9e3025d1fb53bdba8acf*
/.tmp/dc4ce516887f4e0bbaf6201d69ba90bc)
2013-04-16 17:56:07,331 [regionserver8041.cacheFlusher] INFO
org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=210517246,
memsize=39.3m, into tmp file hdfs://hbase-master:8000/hbase
/URL_COUNTERS/*af2760e4d04a9e3025d1fb53bdba8acf*
/.tmp/dc4ce516887f4e0bbaf6201d69ba90bc
2013-04-16 17:56:07,357 [regionserver8041.cacheFlusher] INFO
org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter
type for hdfs://hbase-master:8000/hbase/URL_COUNTERS/*af2760e*
*4d04a9e3025d1fb53bdba8acf*/.tmp/3fa7993dcb294be1bca5e4d7357f4003:
CompoundBloomFilterWriter
2013-04-16 17:56:07,608 [regionserver8041.cacheFlusher] INFO
org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO
DeleteFamily was added to HFile (hdfs://hbase-master:8000/hbase
/URL_COUNTERS/*af2760e4d04a9e3025d1fb53bdba8acf*
/.tmp/3fa7993dcb294be1bca5e4d7357f4003)
2013-04-16 17:56:07,608 [regionserver8041.cacheFlusher] FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
region-server-address,8041,1364993168088: Replay of HLog required
. Forcing server shutdown
DroppedSnapshotException: region: TABLE,ROWKEY,1364317591568.*
af2760e4d04a9e3025d1fb53bdba8acf*.
....
....
...
On Wed, Apr 17, 2013 at 11:47 AM, ramkrishna vasudevan <
[EMAIL PROTECTED]> wrote:

> Seems interesting.  Can  you tell us what are the families and the
> qualifiers available in your schema.
>
> Any other interesting logs that you can see before this?
>
> BTW the version of HBase is also needed?  If we can track it out we can
> then file a JIRA if it is a bug.
>
> Regards
> RAm
>
>
> On Wed, Apr 17, 2013 at 2:00 PM, Amit Sela <[EMAIL PROTECTED]> wrote:
>
> > Hi all,
> >
> > I had a regionserver crushed during counters increment. Looking at the
> > regionserver log I saw:
> >
> > org.apache.hadoop.hbase.DroppedSnapshotException: region: TABLE_NAME,
> > ROW_KEY...at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1472)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1351)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1292)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:406)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:380)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:243)
> >         at java.lang.Thread.run(Thread.java:722)
> > Caused by: java.io.IOException: ScanWildcardColumnTracker.checkColumn ran
> > into a column actually smaller than the previous column: *QUALIFIER*
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.ScanWildcardColumnTracker.checkColumn(ScanWildcardColumnTracker.java:104)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:354)
> >         at
> >
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB