Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Table listed in "list", but not in .META.


Copy link to this message
-
Re: Table listed in "list", but not in .META.
anil gupta 2012-08-11, 04:37
Good to hear that... :) I also had a very bad crash of HBase a few days
ago... at that time had to run hbck repair to fix some errors and to fix
others i had to remove files from /hbase dir. It's a fairly complex system
so sometimes it really hard to nail the problem.

~Anil

On Fri, Aug 10, 2012 at 8:21 PM, Marco Gallotta <[EMAIL PROTECTED]>wrote:

> w00t! I looked in "/hbase/" on the hdfs and found directories referencing
> the stale tables and deleting them worked. It now says 0 inconsistencies.
> Maybe worth adding this to the hbck -repair script? Anyway, thanks everyone
> for the help! Hopefully this stops the sporadic crashes I was seeing.
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> [EMAIL PROTECTED] | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 5:20 PM, Marco Gallotta wrote:
>
> > I never did that, but I mentioned above that all these come from when I
> ^C'ed (i.e. killed) table creation when I was trying to get lzo compression
> working and table creation was hanging. Where would I check for references
> to these tables in ZK?
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog(
> http://twitter.com/marcog)
> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > On Friday 10 August 2012 at 5:15 PM, lars hofhansl wrote:
> >
> > > Did you blow away your cluster once (in the sense of the deleting all
> data and starting from scratch)?This could be left-over data in ZK.
> > >
> > >
> > >
> > > ----- Original Message -----
> > > From: Marco Gallotta <[EMAIL PROTECTED] (mailto:
> [EMAIL PROTECTED])>
> > > To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> > > Cc:
> > > Sent: Friday, August 10, 2012 2:22 PM
> > > Subject: Table listed in "list", but not in .META.
> > >
> > > Hi there
> > >
> > > I have a few tables which show up in a "list" in the shell, but
> produce "table not found" when performing any operation on them. There is
> no reference of them in the .META. table. It seems to be resulting in some
> of the hbase services being killed every so often.
> > >
> > > Here are some logs from master (foo is one of the tables not found):
> > >
> > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> Master server abort: loaded coprocessors are: []
> > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> state=PENDING_OPEN, ts=1344570044277,
> server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 ..
> Cannot transit it to OFFLINE.
> > >
> > >
> > > There are also a number of the following types of error logs:
> > >
> > > 2012-08-09 20:10:04,308 ERROR
> org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
> ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> Received:OPEN for the
> region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are
> already trying to OPEN.
> > >
> > > Any ideas how to find and remove any references to these non-existent
> tables?
> > >
> > > --
> > > Marco Gallotta | Mountain View, California
> > > Software Engineer, Infrastructure | Loki Studios
> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> twitter.com/marcog (http://twitter.com/marcog)
> > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> > >
> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
>
>
--
Thanks & Regards,
Anil Gupta