Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> RegionServer shutdown with ScanWildcardColumnTracker exception


Copy link to this message
-
Re: RegionServer shutdown with ScanWildcardColumnTracker exception
No. It happened in our production environment after running counters
increments every 5 minutes for a few weeks now. I could try to reproduce in
test cluster environment but that would mean running for weeks as well...
but I will keep digging and let you guys know if it happens again or / and
I have more information or insights on the issue.

Thanks.
On Wed, Apr 17, 2013 at 8:18 PM, ramkrishna vasudevan <
[EMAIL PROTECTED]> wrote:

> Is there any testcases that tries to reproduce your issue?
>
> Regards
> Ram
>
>
> On Wed, Apr 17, 2013 at 9:47 PM, ramkrishna vasudevan <
> [EMAIL PROTECTED]> wrote:
>
> > There is a hint mechanism available when scanning happens.  But i dont
> > think there should be much of difference between a scan that happens
> during
> > flush and the normal scan.
> >
> > Will look thro the code and come back on this.
> >
> > Regards
> > RAm
> >
> >
> > On Wed, Apr 17, 2013 at 9:40 PM, Amit Sela <[EMAIL PROTECTED]> wrote:
> >
> >> No, no encoding.
> >>
> >>
> >> On Wed, Apr 17, 2013 at 6:56 PM, ramkrishna vasudevan <
> >> [EMAIL PROTECTED]> wrote:
> >>
> >> > @Lars
> >> > You have any suggestions on this?
> >> >
> >> > @Amit
> >> > You have any Encoder enabled like the Prefix Encoding stuff?
> >> > There was one optimization added recently but that is not in 0.94.2
> >> >
> >> > Regards
> >> > Ram
> >> >
> >> >
> >> > On Wed, Apr 17, 2013 at 5:17 PM, Amit Sela <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> > > I scanned over this counter with and without column specification
> and
> >> all
> >> > > looks OK now.
> >> > > I have no CPs in this table.
> >> > > Is there some kind of a hint mechanism in HBase' internal scan ?
> >> because
> >> > > it's weird that ScanWildcardColumnTracker.checkColumn says that
> >> column is
> >> > > smaller than previous column: *imprersions_ALL_2013041617*. there
> is
> >> no
> >> > > imprersions only impressions and r is indeed smaller than s, could
> it
> >> be
> >> > > some kind of hint bug ? I don't think I know enough of HBase
> >> internals to
> >> > > fully understand that...
> >> > >
> >> > >
> >> > >
> >> > > On Wed, Apr 17, 2013 at 1:42 PM, ramkrishna vasudevan <
> >> > > [EMAIL PROTECTED]> wrote:
> >> > >
> >> > > > Hi Amit
> >> > > >
> >> > > > Checking the code this is possible when the qualifiers are not
> >> sorted.
> >> > >  Do
> >> > > > you have any CPs in your path which tries to play with the KVs?
> >> > > >
> >> > > > Seems to be a very weird thing.
> >> > > > Can you try doing a scan on the KV just before this happens.  That
> >> will
> >> > > tel
> >> > > > you the existing kvs that are present.
> >> > > >
> >> > > > Even now if you can have the cluster you can try scanning for the
> >> > region
> >> > > > for which the flush happened.  That will give us some more info.
> >> > > >
> >> > > > Regards
> >> > > > Ram
> >> > > >
> >> > > >
> >> > > > On Wed, Apr 17, 2013 at 2:36 PM, Amit Sela <[EMAIL PROTECTED]>
> >> > wrote:
> >> > > >
> >> > > > > The cluster runs Hadoop 1.0.4 and HBase 0.94.2
> >> > > > >
> >> > > > > I have three families in this table: weekly, daily, hourly. each
> >> > family
> >> > > > has
> >> > > > > the following qualifiers:
> >> > > > > Weekly - impressions_{countrycode}_{week#} - country code is 0,
> 1
> >> or
> >> > > ALL
> >> > > > > (aggregation of both 0 and 1)
> >> > > > > Daily and hourly are the same but with yyyyMMdd and yyyyMMddhh
> >> > > > > respectively.
> >> > > > >
> >> > > > > Just before the exception the regionserver StoreFile executes
> the
> >> > > > > following:
> >> > > > >
> >> > > > > 2013-04-16 17:56:06,769 [regionserver8041.cacheFlusher] INFO
> >> > > > > org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family
> >> Bloom
> >> > > > filter
> >> > > > > type for hdfs://
> >> > > > > hadoop-master.infolinks.com:8000/hbase/URL_COUNTERS/af2760e
> >> > > > > 4d04a9e3025d1fb53bdba8acf/.tmp/dc4ce516887f4e0bbaf6201d69ba90bc:
> >> > > > > CompoundBloomFilterWriter
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB