Hi there

I have a few tables which show up in a "list" in the shell, but produce "table not found" when performing any operation on them. There is no reference of them in the .META. table. It seems to be resulting in some of the hbase services being killed every so often.

Here are some logs from master (foo is one of the tables not found):

2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. state=PENDING_OPEN, ts=1344570044277, server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 .. Cannot transit it to OFFLINE.
There are also a number of the following types of error logs:

2012-08-09 20:10:04,308 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are already trying to OPEN.

Any ideas how to find and remove any references to these non-existent tables?

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

Hi Marco,

Did anything disastrous happen to cluster?
Can you try using hbck utility of HBase.
Run: 'hbase hbck -help' to get all the available options.

~Anil

On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <[EMAIL PROTECTED]>wrote:

> Hi there
>
> I have a few tables which show up in a "list" in the shell, but produce
> "table not found" when performing any operation on them. There is no
> reference of them in the .META. table. It seems to be resulting in some of
> the hbase services being killed every so often.
>
> Here are some logs from master (foo is one of the tables not found):
>
> 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> Master server abort: loaded coprocessors are: []
> 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> state=PENDING_OPEN, ts=1344570044277,
> server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 ..
> Cannot transit it to OFFLINE.
>
>
> There are also a number of the following types of error logs:
>
> 2012-08-09 20:10:04,308 ERROR
> org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
> ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> Received:OPEN for the
> region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are
> already trying to OPEN.
>
> Any ideas how to find and remove any references to these non-existent
> tables?
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> [EMAIL PROTECTED] | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
--
Thanks & Regards,
Anil Gupta

Here's the output from hbck -details: http://pastebin.com/ZxVZEctY

Extract:

6 inconsistencies detected.
Status: INCONSISTENT

6 is the number of tables that appear in "list" but cannot be operated on (which btw, includes not being able to run disable/drop on them - both ops say table not found). I also just noticed "foo" does not occur in a table list, although I did create it at one point but was able to clear it from .META. when it also was reporting table not found when trying to disable/drop it. All these come from when I ^C'ed (i.e. killed) table creation when I was trying to get lzo compression working and table creation was hanging.

Is there any way to repair this? I see hbck has repair options, but I want to proceed with caution.

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:

> Hi Marco,
>
> Did anything disastrous happen to cluster?
> Can you try using hbck utility of HBase.
> Run: 'hbase hbck -help' to get all the available options.
>
> ~Anil
>
> On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>wrote:
>
> > Hi there
> >
> > I have a few tables which show up in a "list" in the shell, but produce
> > "table not found" when performing any operation on them. There is no
> > reference of them in the .META. table. It seems to be resulting in some of
> > the hbase services being killed every so often.
> >
> > Here are some logs from master (foo is one of the tables not found):
> >
> > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> > Master server abort: loaded coprocessors are: []
> > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> > Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > state=PENDING_OPEN, ts=1344570044277,
> > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 ..
> > Cannot transit it to OFFLINE.
> >
> >
> > There are also a number of the following types of error logs:
> >
> > 2012-08-09 20:10:04,308 ERROR
> > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
> > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
> > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > Received:OPEN for the
> > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are
> > already trying to OPEN.
> >
> > Any ideas how to find and remove any references to these non-existent
> > tables?
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
>

Are you running a distributed cluster?
If yes, do you have localhost in /etc/hosts file?

You are getting reference to localhost in hbck output:
ERROR: Region { meta => null, hdfs =>
hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
deployed =>  } on HDFS, but not listed in META or deployed on any region
server

~Anil

On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <[EMAIL PROTECTED]>wrote:

> Here's the output from hbck -details: http://pastebin.com/ZxVZEctY
>
> Extract:
>
> 6 inconsistencies detected.
> Status: INCONSISTENT
>
> 6 is the number of tables that appear in "list" but cannot be operated on
> (which btw, includes not being able to run disable/drop on them - both ops
> say table not found). I also just noticed "foo" does not occur in a table
> list, although I did create it at one point but was able to clear it from
> .META. when it also was reporting table not found when trying to
> disable/drop it. All these come from when I ^C'ed (i.e. killed) table
> creation when I was trying to get lzo compression working and table
> creation was hanging.
>
> Is there any way to repair this? I see hbck has repair options, but I want
> to proceed with caution.
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> [EMAIL PROTECTED] | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>
> > Hi Marco,
> >
> > Did anything disastrous happen to cluster?
> > Can you try using hbck utility of HBase.
> > Run: 'hbase hbck -help' to get all the available options.
> >
> > ~Anil
> >
> > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <[EMAIL PROTECTED](mailto:
> [EMAIL PROTECTED])>wrote:
> >
> > > Hi there
> > >
> > > I have a few tables which show up in a "list" in the shell, but produce
> > > "table not found" when performing any operation on them. There is no
> > > reference of them in the .META. table. It seems to be resulting in
> some of
> > > the hbase services being killed every so often.
> > >
> > > Here are some logs from master (foo is one of the tables not found):
> > >
> > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > Master server abort: loaded coprocessors are: []
> > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > state=PENDING_OPEN, ts=1344570044277,
> > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> ..
> > > Cannot transit it to OFFLINE.
> > >
> > >
> > > There are also a number of the following types of error logs:
> > >
> > > 2012-08-09 20:10:04,308 ERROR
> > > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
> > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
> > >
> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > > Received:OPEN for the
> > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we
> are
> > > already trying to OPEN.
> > >
> > > Any ideas how to find and remove any references to these non-existent
> > > tables?
> > >
> > > --
> > > Marco Gallotta | Mountain View, California
> > > Software Engineer, Infrastructure | Loki Studios
> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> twitter.com/marcog (http://twitter.com/marcog)
> > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> > >
> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
> >
>
>
>
--
Thanks & Regards,
Anil Gupta

It's not a distributed cluster. I'm not processing enough data yet. So the reference to localhost is correct.

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:

> Are you running a distributed cluster?
> If yes, do you have localhost in /etc/hosts file?
>
> You are getting reference to localhost in hbck output:
> ERROR: Region { meta => null, hdfs =>
> hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> deployed => } on HDFS, but not listed in META or deployed on any region
> server
>
> ~Anil
>
> On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>wrote:
>
> > Here's the output from hbck -details: http://pastebin.com/ZxVZEctY
> >
> > Extract:
> >
> > 6 inconsistencies detected.
> > Status: INCONSISTENT
> >
> > 6 is the number of tables that appear in "list" but cannot be operated on
> > (which btw, includes not being able to run disable/drop on them - both ops
> > say table not found). I also just noticed "foo" does not occur in a table
> > list, although I did create it at one point but was able to clear it from
> > .META. when it also was reporting table not found when trying to
> > disable/drop it. All these come from when I ^C'ed (i.e. killed) table
> > creation when I was trying to get lzo compression working and table
> > creation was hanging.
> >
> > Is there any way to repair this? I see hbck has repair options, but I want
> > to proceed with caution.
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> >
> > > Hi Marco,
> > >
> > > Did anything disastrous happen to cluster?
> > > Can you try using hbck utility of HBase.
> > > Run: 'hbase hbck -help' to get all the available options.
> > >
> > > ~Anil
> > >
> > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))>wrote:
> > >
> > > > Hi there
> > > >
> > > > I have a few tables which show up in a "list" in the shell, but produce
> > > > "table not found" when performing any operation on them. There is no
> > > > reference of them in the .META. table. It seems to be resulting in
> > > >
> > >
> > >
> >
> > some of
> > > > the hbase services being killed every so often.
> > > >
> > > > Here are some logs from master (foo is one of the tables not found):
> > > >
> > > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > > Master server abort: loaded coprocessors are: []
> > > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > > Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > state=PENDING_OPEN, ts=1344570044277,
> > > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > >
> > >
> >
> > ..
> > > > Cannot transit it to OFFLINE.
> > > >
> > > >
> > > > There are also a number of the following types of error logs:
> > > >
> > > > 2012-08-09 20:10:04,308 ERROR
> > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
> > > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
> > > >
> > >
> >
> > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > > > Received:OPEN for the
> > > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we
> > > >
> > >
> >
> > are
> > > > already trying to OPEN.

Could you please share your /etc/hosts file??Meantime, do a manual
compaction and see if ti works.

Regards,
    Mohammad Tariq
On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <[EMAIL PROTECTED]> wrote:
> It's not a distributed cluster. I'm not processing enough data yet. So the reference to localhost is correct.
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> [EMAIL PROTECTED] | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
>
>> Are you running a distributed cluster?
>> If yes, do you have localhost in /etc/hosts file?
>>
>> You are getting reference to localhost in hbck output:
>> ERROR: Region { meta => null, hdfs =>
>> hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
>> deployed => } on HDFS, but not listed in META or deployed on any region
>> server
>>
>> ~Anil
>>
>> On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>wrote:
>>
>> > Here's the output from hbck -details: http://pastebin.com/ZxVZEctY
>> >
>> > Extract:
>> >
>> > 6 inconsistencies detected.
>> > Status: INCONSISTENT
>> >
>> > 6 is the number of tables that appear in "list" but cannot be operated on
>> > (which btw, includes not being able to run disable/drop on them - both ops
>> > say table not found). I also just noticed "foo" does not occur in a table
>> > list, although I did create it at one point but was able to clear it from
>> > .META. when it also was reporting table not found when trying to
>> > disable/drop it. All these come from when I ^C'ed (i.e. killed) table
>> > creation when I was trying to get lzo compression working and table
>> > creation was hanging.
>> >
>> > Is there any way to repair this? I see hbck has repair options, but I want
>> > to proceed with caution.
>> >
>> > --
>> > Marco Gallotta | Mountain View, California
>> > Software Engineer, Infrastructure | Loki Studios
>> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
>> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
>> >
>> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> >
>> >
>> > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>> >
>> > > Hi Marco,
>> > >
>> > > Did anything disastrous happen to cluster?
>> > > Can you try using hbck utility of HBase.
>> > > Run: 'hbase hbck -help' to get all the available options.
>> > >
>> > > ~Anil
>> > >
>> > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
>> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))>wrote:
>> > >
>> > > > Hi there
>> > > >
>> > > > I have a few tables which show up in a "list" in the shell, but produce
>> > > > "table not found" when performing any operation on them. There is no
>> > > > reference of them in the .META. table. It seems to be resulting in
>> > > >
>> > >
>> > >
>> >
>> > some of
>> > > > the hbase services being killed every so often.
>> > > >
>> > > > Here are some logs from master (foo is one of the tables not found):
>> > > >
>> > > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
>> > > > Master server abort: loaded coprocessors are: []
>> > > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
>> > > > Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>> > > > state=PENDING_OPEN, ts=1344570044277,
>> > > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>> > > >
>> > >
>> >
>> > ..
>> > > > Cannot transit it to OFFLINE.
>> > > >
>> > > >
>> > > > There are also a number of the following types of error logs:
>> > > >
>> > > > 2012-08-09 20:10:04,308 ERROR
>> > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
>> > > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to

Is it a standalone installation or pseudo-distributed?
I faced a similar problem a few days back in a distributed cluster and used
hbck -repair option. You might give it a try.

~Anil

On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:

> Could you please share your /etc/hosts file??Meantime, do a manual
> compaction and see if ti works.
>
> Regards,
>     Mohammad Tariq
>
>
> On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <[EMAIL PROTECTED]>
> wrote:
> > It's not a distributed cluster. I'm not processing enough data yet. So
> the reference to localhost is correct.
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta | twitter.com/marcog
> > [EMAIL PROTECTED] | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> >
> >> Are you running a distributed cluster?
> >> If yes, do you have localhost in /etc/hosts file?
> >>
> >> You are getting reference to localhost in hbck output:
> >> ERROR: Region { meta => null, hdfs =>
> >> hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> >> deployed => } on HDFS, but not listed in META or deployed on any region
> >> server
> >>
> >> ~Anil
> >>
> >> On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <[EMAIL PROTECTED](mailto:
> [EMAIL PROTECTED])>wrote:
> >>
> >> > Here's the output from hbck -details: http://pastebin.com/ZxVZEctY
> >> >
> >> > Extract:
> >> >
> >> > 6 inconsistencies detected.
> >> > Status: INCONSISTENT
> >> >
> >> > 6 is the number of tables that appear in "list" but cannot be
> operated on
> >> > (which btw, includes not being able to run disable/drop on them -
> both ops
> >> > say table not found). I also just noticed "foo" does not occur in a
> table
> >> > list, although I did create it at one point but was able to clear it
> from
> >> > .META. when it also was reporting table not found when trying to
> >> > disable/drop it. All these come from when I ^C'ed (i.e. killed) table
> >> > creation when I was trying to get lzo compression working and table
> >> > creation was hanging.
> >> >
> >> > Is there any way to repair this? I see hbck has repair options, but I
> want
> >> > to proceed with caution.
> >> >
> >> > --
> >> > Marco Gallotta | Mountain View, California
> >> > Software Engineer, Infrastructure | Loki Studios
> >> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> twitter.com/marcog (http://twitter.com/marcog)
> >> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650)
> 417-3313
> >> >
> >> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >> >
> >> >
> >> > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> >> >
> >> > > Hi Marco,
> >> > >
> >> > > Did anything disastrous happen to cluster?
> >> > > Can you try using hbck utility of HBase.
> >> > > Run: 'hbase hbck -help' to get all the available options.
> >> > >
> >> > > ~Anil
> >> > >
> >> > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
> >> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))>wrote:
> >> > >
> >> > > > Hi there
> >> > > >
> >> > > > I have a few tables which show up in a "list" in the shell, but
> produce
> >> > > > "table not found" when performing any operation on them. There is
> no
> >> > > > reference of them in the .META. table. It seems to be resulting in
> >> > > >
> >> > >
> >> > >
> >> >
> >> > some of
> >> > > > the hbase services being killed every so often.
> >> > > >
> >> > > > Here are some logs from master (foo is one of the tables not
> found):
> >> > > >
> >> > > > 2012-08-09 20:40:44,301 FATAL
> org.apache.hadoop.hbase.master.HMaster:
> >> > > > Master server abort: loaded coprocessors are: []
> >> > > > 2012-08-09 20:40:44,301 FATAL
> org.apache.hadoop.hbase.master.HMaster:
> >> > > > Unexpected state :
> foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.

Thanks & Regards,
Anil Gupta

It's a pseudo-distributed cluster, as I plan to add more nodes as we start gathering more data.

I get the following error when running hbck -repair, and then it stalls:

12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking after fix...
Version: 0.94.0
12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from HDFS...
Exception in thread "main" java.util.concurrent.RejectedExecutionException
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
at org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:

> Is it a standalone installation or pseudo-distributed?
> I faced a similar problem a few days back in a distributed cluster and used
> hbck -repair option. You might give it a try.
>
> ~Anil
>
> On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
>
> > Could you please share your /etc/hosts file??Meantime, do a manual
> > compaction and see if ti works.
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
> > wrote:
> > > It's not a distributed cluster. I'm not processing enough data yet. So
> >
> > the reference to localhost is correct.
> > >
> > > --
> > > Marco Gallotta | Mountain View, California
> > > Software Engineer, Infrastructure | Loki Studios
> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> > >
> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > >
> > >
> > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > >
> > > > Are you running a distributed cluster?
> > > > If yes, do you have localhost in /etc/hosts file?
> > > >
> > > > You are getting reference to localhost in hbck output:
> > > > ERROR: Region { meta => null, hdfs =>
> > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > > deployed => } on HDFS, but not listed in META or deployed on any region
> > > > server
> > > >
> > > > ~Anil
> > > >
> > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))>wrote:
> > > >
> > > > > Here's the output from hbck -details: http://pastebin.com/ZxVZEctY
> > > > >
> > > > > Extract:
> > > > >
> > > > > 6 inconsistencies detected.
> > > > > Status: INCONSISTENT
> > > > >
> > > > > 6 is the number of tables that appear in "list" but cannot be
> > operated on
> > > > > (which btw, includes not being able to run disable/drop on them -
> > > >
> > >
> >
> > both ops
> > > > > say table not found). I also just noticed "foo" does not occur in a
> > > >
> > >
> >
> > table
> > > > > list, although I did create it at one point but was able to clear it
> > > >
> > >
> >
> > from
> > > > > .META. when it also was reporting table not found when trying to
> > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed) table
> > > > > creation when I was trying to get lzo compression working and table

Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
rebooted your machined then you will have problems.

On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <[EMAIL PROTECTED]>wrote:

> It's a pseudo-distributed cluster, as I plan to add more nodes as we start
> gathering more data.
>
> I get the following error when running hbck -repair, and then it stalls:
>
> 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
> after fix...
> Version: 0.94.0
> 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
> 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
> HDFS...
> Exception in thread "main" java.util.concurrent.RejectedExecutionException
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
>
>
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> [EMAIL PROTECTED] | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>
> > Is it a standalone installation or pseudo-distributed?
> > I faced a similar problem a few days back in a distributed cluster and
> used
> > hbck -repair option. You might give it a try.
> >
> > ~Anil
> >
> > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <[EMAIL PROTECTED](mailto:
> [EMAIL PROTECTED])> wrote:
> >
> > > Could you please share your /etc/hosts file??Meantime, do a manual
> > > compaction and see if ti works.
> > >
> > > Regards,
> > > Mohammad Tariq
> > >
> > >
> > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <[EMAIL PROTECTED](mailto:
> [EMAIL PROTECTED])>
> > > wrote:
> > > > It's not a distributed cluster. I'm not processing enough data yet.
> So
> > >
> > > the reference to localhost is correct.
> > > >
> > > > --
> > > > Marco Gallotta | Mountain View, California
> > > > Software Engineer, Infrastructure | Loki Studios
> > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> twitter.com/marcog (http://twitter.com/marcog)
> > > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650)
> 417-3313
> > > >
> > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > >
> > > >
> > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > > >
> > > > > Are you running a distributed cluster?
> > > > > If yes, do you have localhost in /etc/hosts file?
> > > > >
> > > > > You are getting reference to localhost in hbck output:
> > > > > ERROR: Region { meta => null, hdfs =>
> > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > > > deployed => } on HDFS, but not listed in META or deployed on any
> region
> > > > > server
> > > > >
> > > > > ~Anil
> > > > >
> > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
> > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))>wrote:
> > > > >
> > > > > > Here's the output from hbck -details:
> http://pastebin.com/ZxVZEctY
> > > > > >
> > > > > > Extract:
> > > > > >
> > > > > > 6 inconsistencies detected.
> > > > > > Status: INCONSISTENT
> > > > > >
> > > > > > 6 is the number of tables that appear in "list" but cannot be
> > > operated on
> > > > > > (which btw, includes not being able to run disable/drop on them -
> > > > >
> > > >
> > >
> > > both ops
Thanks & Regards,
Anil Gupta

It's in /var which is persistent across reboots.

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:

> Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
> rebooted your machined then you will have problems.
>
> On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>wrote:
>
> > It's a pseudo-distributed cluster, as I plan to add more nodes as we start
> > gathering more data.
> >
> > I get the following error when running hbck -repair, and then it stalls:
> >
> > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
> > after fix...
> > Version: 0.94.0
> > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
> > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
> > HDFS...
> > Exception in thread "main" java.util.concurrent.RejectedExecutionException
> > at
> > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
> > at
> > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> > at
> > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
> > at
> > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
> > at
> > org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
> > at
> > org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
> > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
> > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
> >
> >
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
> >
> > > Is it a standalone installation or pseudo-distributed?
> > > I faced a similar problem a few days back in a distributed cluster and
> > >
> >
> > used
> > > hbck -repair option. You might give it a try.
> > >
> > > ~Anil
> > >
> > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))> wrote:
> > >
> > > > Could you please share your /etc/hosts file??Meantime, do a manual
> > > > compaction and see if ti works.
> > > >
> > > > Regards,
> > > > Mohammad Tariq
> > > >
> > > >
> > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))>
> > > > wrote:
> > > > > It's not a distributed cluster. I'm not processing enough data yet.
> > > >
> > > >
> > >
> >
> > So
> > > >
> > > > the reference to localhost is correct.
> > > > >
> > > > > --
> > > > > Marco Gallotta | Mountain View, California
> > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > >
> > > >
> > > >
> > >
> >
> > twitter.com/marcog (http://twitter.com/marcog)
> > > > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650)
> > > >
> > >
> >
> > 417-3313
> > > > >
> > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > >
> > > > >
> > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > > > >
> > > > > > Are you running a distributed cluster?
> > > > > > If yes, do you have localhost in /etc/hosts file?
> > > > > >
> > > > > > You are getting reference to localhost in hbck output:

What about fs.default.name?????

Regards,
    Mohammad Tariq
On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <[EMAIL PROTECTED]> wrote:
> It's in /var which is persistent across reboots.
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> [EMAIL PROTECTED] | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
>
>> Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
>> rebooted your machined then you will have problems.
>>
>> On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>wrote:
>>
>> > It's a pseudo-distributed cluster, as I plan to add more nodes as we start
>> > gathering more data.
>> >
>> > I get the following error when running hbck -repair, and then it stalls:
>> >
>> > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
>> > after fix...
>> > Version: 0.94.0
>> > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
>> > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
>> > HDFS...
>> > Exception in thread "main" java.util.concurrent.RejectedExecutionException
>> > at
>> > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>> > at
>> > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
>> > at
>> > org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
>> > at
>> > org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
>> > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
>> > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
>> >
>> >
>> >
>> > --
>> > Marco Gallotta | Mountain View, California
>> > Software Engineer, Infrastructure | Loki Studios
>> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
>> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
>> >
>> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> >
>> >
>> > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>> >
>> > > Is it a standalone installation or pseudo-distributed?
>> > > I faced a similar problem a few days back in a distributed cluster and
>> > >
>> >
>> > used
>> > > hbck -repair option. You might give it a try.
>> > >
>> > > ~Anil
>> > >
>> > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
>> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))> wrote:
>> > >
>> > > > Could you please share your /etc/hosts file??Meantime, do a manual
>> > > > compaction and see if ti works.
>> > > >
>> > > > Regards,
>> > > > Mohammad Tariq
>> > > >
>> > > >
>> > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
>> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))>
>> > > > wrote:
>> > > > > It's not a distributed cluster. I'm not processing enough data yet.
>> > > >
>> > > >
>> > >
>> >
>> > So
>> > > >
>> > > > the reference to localhost is correct.
>> > > > >
>> > > > > --
>> > > > > Marco Gallotta | Mountain View, California
>> > > > > Software Engineer, Infrastructure | Loki Studios
>> > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>> > > > >
>> > > >
>> > > >
>> > >
>> >
>> > twitter.com/marcog (http://twitter.com/marcog)
>> > > > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650)
>> > > >
>> > >
>> >
>> > 417-3313
>> > > > >
>> > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > > > >
>> > > > >
>> > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:

 <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 4:43 PM, Mohammad Tariq wrote:

> What about fs.default.name (http://fs.default.name)?????
>
> Regards,
> Mohammad Tariq
>
>
> On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > It's in /var which is persistent across reboots.
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
> >
> > > Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
> > > rebooted your machined then you will have problems.
> > >
> > > On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>wrote:
> > >
> > > > It's a pseudo-distributed cluster, as I plan to add more nodes as we start
> > > > gathering more data.
> > > >
> > > > I get the following error when running hbck -repair, and then it stalls:
> > > >
> > > > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
> > > > after fix...
> > > > Version: 0.94.0
> > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
> > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
> > > > HDFS...
> > > > Exception in thread "main" java.util.concurrent.RejectedExecutionException
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
> > > > at
> > > > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
> > > > at
> > > > org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
> > > > at
> > > > org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
> > > > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
> > > > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
> > > >
> > > >
> > > >
> > > > --
> > > > Marco Gallotta | Mountain View, California
> > > > Software Engineer, Infrastructure | Loki Studios
> > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> > > >
> > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > >
> > > >
> > > > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
> > > >
> > > > > Is it a standalone installation or pseudo-distributed?
> > > > > I faced a similar problem a few days back in a distributed cluster and
> > > > >
> > > >
> > > >
> > > > used
> > > > > hbck -repair option. You might give it a try.
> > > > >
> > > > > ~Anil
> > > > >
> > > > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
> > > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))> wrote:
> > > > >
> > > > > > Could you please share your /etc/hosts file??Meantime, do a manual
> > > > > > compaction and see if ti works.
> > > > > >
> > > > > > Regards,
> > > > > > Mohammad Tariq
> > > > > >
> > > > > >
> > > > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:

This is pretty strange. I mean everything seems to be in place, but we
are stuck. Please make a check once if your Hdfs is in safemode.

Regards,
    Mohammad Tariq
On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> What about fs.default.name?????
>
> Regards,
>     Mohammad Tariq
>
>
> On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <[EMAIL PROTECTED]> wrote:
>> It's in /var which is persistent across reboots.
>>
>> --
>> Marco Gallotta | Mountain View, California
>> Software Engineer, Infrastructure | Loki Studios
>> fb.me/marco.gallotta | twitter.com/marcog
>> [EMAIL PROTECTED] | +1 (650) 417-3313
>>
>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>
>>
>> On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
>>
>>> Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
>>> rebooted your machined then you will have problems.
>>>
>>> On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>wrote:
>>>
>>> > It's a pseudo-distributed cluster, as I plan to add more nodes as we start
>>> > gathering more data.
>>> >
>>> > I get the following error when running hbck -repair, and then it stalls:
>>> >
>>> > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
>>> > after fix...
>>> > Version: 0.94.0
>>> > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
>>> > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
>>> > HDFS...
>>> > Exception in thread "main" java.util.concurrent.RejectedExecutionException
>>> > at
>>> > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
>>> > at
>>> > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>>> > at
>>> > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>>> > at
>>> > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
>>> > at
>>> > org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
>>> > at
>>> > org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
>>> > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
>>> > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
>>> >
>>> >
>>> >
>>> > --
>>> > Marco Gallotta | Mountain View, California
>>> > Software Engineer, Infrastructure | Loki Studios
>>> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
>>> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
>>> >
>>> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>> >
>>> >
>>> > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>>> >
>>> > > Is it a standalone installation or pseudo-distributed?
>>> > > I faced a similar problem a few days back in a distributed cluster and
>>> > >
>>> >
>>> > used
>>> > > hbck -repair option. You might give it a try.
>>> > >
>>> > > ~Anil
>>> > >
>>> > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
>>> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))> wrote:
>>> > >
>>> > > > Could you please share your /etc/hosts file??Meantime, do a manual
>>> > > > compaction and see if ti works.
>>> > > >
>>> > > > Regards,
>>> > > > Mohammad Tariq
>>> > > >
>>> > > >
>>> > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
>>> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))>
>>> > > > wrote:
>>> > > > > It's not a distributed cluster. I'm not processing enough data yet.
>>> > > >
>>> > > >
>>> > >
>>> >
>>> > So
>>> > > >
>>> > > > the reference to localhost is correct.
>>> > > > >
>>> > > > > --
>>> > > > > Marco Gallotta | Mountain View, California
>>> > > > > Software Engineer, Infrastructure | Loki Studios
>>> > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>

How do you check that?

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:

> This is pretty strange. I mean everything seems to be in place, but we
> are stuck. Please make a check once if your Hdfs is in safemode.
>
> Regards,
> Mohammad Tariq
>
>
> On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > What about fs.default.name (http://fs.default.name)?????
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> > On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > It's in /var which is persistent across reboots.
> > >
> > > --
> > > Marco Gallotta | Mountain View, California
> > > Software Engineer, Infrastructure | Loki Studios
> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> > >
> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > >
> > >
> > > On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
> > >
> > > > Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
> > > > rebooted your machined then you will have problems.
> > > >
> > > > On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>wrote:
> > > >
> > > > > It's a pseudo-distributed cluster, as I plan to add more nodes as we start
> > > > > gathering more data.
> > > > >
> > > > > I get the following error when running hbck -repair, and then it stalls:
> > > > >
> > > > > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
> > > > > after fix...
> > > > > Version: 0.94.0
> > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
> > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
> > > > > HDFS...
> > > > > Exception in thread "main" java.util.concurrent.RejectedExecutionException
> > > > > at
> > > > > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
> > > > > at
> > > > > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> > > > > at
> > > > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
> > > > > at
> > > > > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
> > > > > at
> > > > > org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
> > > > > at
> > > > > org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
> > > > > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
> > > > > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Marco Gallotta | Mountain View, California
> > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > > > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> > > > >
> > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > >
> > > > >
> > > > > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
> > > > >
> > > > > > Is it a standalone installation or pseudo-distributed?
> > > > > > I faced a similar problem a few days back in a distributed cluster and
> > > > > >
> > > > >
> > > > >
> > > > > used
> > > > > > hbck -repair option. You might give it a try.
> > > > > >
> > > > > > ~Anil
> > > > > >
> > > > > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
> > > > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))> wrote:

You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
point your web browser to http://localhost:9000. It'll show all the
details of your HDFS.

Regards,
    Mohammad Tariq
On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <[EMAIL PROTECTED]> wrote:
> How do you check that?
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> [EMAIL PROTECTED] | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
>
>> This is pretty strange. I mean everything seems to be in place, but we
>> are stuck. Please make a check once if your Hdfs is in safemode.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>> On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
>> > What about fs.default.name (http://fs.default.name)?????
>> >
>> > Regards,
>> > Mohammad Tariq
>> >
>> >
>> > On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
>> > > It's in /var which is persistent across reboots.
>> > >
>> > > --
>> > > Marco Gallotta | Mountain View, California
>> > > Software Engineer, Infrastructure | Loki Studios
>> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
>> > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
>> > >
>> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > >
>> > >
>> > > On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
>> > >
>> > > > Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
>> > > > rebooted your machined then you will have problems.
>> > > >
>> > > > On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>wrote:
>> > > >
>> > > > > It's a pseudo-distributed cluster, as I plan to add more nodes as we start
>> > > > > gathering more data.
>> > > > >
>> > > > > I get the following error when running hbck -repair, and then it stalls:
>> > > > >
>> > > > > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
>> > > > > after fix...
>> > > > > Version: 0.94.0
>> > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
>> > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
>> > > > > HDFS...
>> > > > > Exception in thread "main" java.util.concurrent.RejectedExecutionException
>> > > > > at
>> > > > > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
>> > > > > at
>> > > > > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>> > > > > at
>> > > > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>> > > > > at
>> > > > > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
>> > > > > at
>> > > > > org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
>> > > > > at
>> > > > > org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
>> > > > > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
>> > > > > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Marco Gallotta | Mountain View, California
>> > > > > Software Engineer, Infrastructure | Loki Studios
>> > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
>> > > > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
>> > > > >
>> > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > > > >
>> > > > >
>> > > > > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>> > > > >
>> > > > > > Is it a standalone installation or pseudo-distributed?
>> > > > > > I faced a similar problem a few days back in a distributed cluster and

If your HDFS is in safemode you'll get something like this :

cluster@ubuntu:~/hadoop-1.0.3$ bin/hadoop dfsadmin -report
Safe mode is ON
Configured Capacity: 31111143424 (28.97 GB)
Present Capacity: 5309755392 (4.95 GB)
DFS Remaining: 4799320064 (4.47 GB)
DFS Used: 510435328 (486.79 MB)
DFS Used%: 9.61%
Under replicated blocks: 1
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)

Name: 127.0.0.1:50010
Decommission Status : Normal
Configured Capacity: 31111143424 (28.97 GB)
DFS Used: 510435328 (486.79 MB)
Non DFS Used: 25801388032 (24.03 GB)
DFS Remaining: 4799320064(4.47 GB)
DFS Used%: 1.64%
DFS Remaining%: 15.43%
Last contact: Sat Aug 11 05:19:18 IST 2012

See the line in red color.

Regards,
    Mohammad Tariq
On Sat, Aug 11, 2012 at 5:20 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
> point your web browser to http://localhost:9000. It'll show all the
> details of your HDFS.
>
> Regards,
>     Mohammad Tariq
>
>
> On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <[EMAIL PROTECTED]>
wrote:
>> How do you check that?
>>
>> --
>> Marco Gallotta | Mountain View, California
>> Software Engineer, Infrastructure | Loki Studios
>> fb.me/marco.gallotta | twitter.com/marcog
>> [EMAIL PROTECTED] | +1 (650) 417-3313
>>
>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>
>>
>> On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
>>
>>> This is pretty strange. I mean everything seems to be in place, but we
>>> are stuck. Please make a check once if your Hdfs is in safemode.
>>>
>>> Regards,
>>> Mohammad Tariq
>>>
>>>
>>> On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <[EMAIL PROTECTED](mailto:
[EMAIL PROTECTED])> wrote:
>>> > What about fs.default.name (http://fs.default.name)?????
>>> >
>>> > Regards,
>>> > Mohammad Tariq
>>> >
>>> >
>>> > On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <[EMAIL PROTECTED](mailto:
[EMAIL PROTECTED])> wrote:
>>> > > It's in /var which is persistent across reboots.
>>> > >
>>> > > --
>>> > > Marco Gallotta | Mountain View, California
>>> > > Software Engineer, Infrastructure | Loki Studios
>>> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
twitter.com/marcog (http://twitter.com/marcog)
>>> > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650)
417-3313
>>> > >
>>> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>> > >
>>> > >
>>> > > On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
>>> > >
>>> > > > Where are you storing your hdfs data? Is it /tmp? If it's /tmp
and you have
>>> > > > rebooted your machined then you will have problems.
>>> > > >
>>> > > > On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <
[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>wrote:
>>> > > >
>>> > > > > It's a pseudo-distributed cluster, as I plan to add more nodes
as we start
>>> > > > > gathering more data.
>>> > > > >
>>> > > > > I get the following error when running hbck -repair, and then
it stalls:
>>> > > > >
>>> > > > > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before
re-checking
>>> > > > > after fix...
>>> > > > > Version: 0.94.0
>>> > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
>>> > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo
from
>>> > > > > HDFS...
>>> > > > > Exception in thread "main"
java.util.concurrent.RejectedExecutionException
>>> > > > > at
>>> > > > >
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
>>> > > > > at
>>> > > > >
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>>> > > > > at
>>> > > > >
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>>> > > > > at
>>> > > > >
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
>>> > > > > at
>>> > > > >
org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
twitter.com/marcog (http://twitter.com/marcog)
417-3313
cluster and
[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
manual
[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
data yet.
(650)
hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
deployed on any
cannot be
disable/drop on them -
not occur
able to
when trying to
(i.e. killed)
working and
repair options,
+1 (650)
options.
))>wrote:
the shell,
on them.
to be
tables not
server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
error logs:
Failed
org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
to these
|
| +1
http://www.sparrowmailapp.com/?sig)

Nope, not in safe mode. Gar, this is going nowhere. :/ Thanks for the help so far though!

Configured Capacity: 211474616320 (196.95 GB)
Present Capacity: 169354764288 (157.72 GB)
DFS Remaining: 162401554432 (151.25 GB)
DFS Used: 6953209856 (6.48 GB)
DFS Used%: 4.11%
Under replicated blocks: 436
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)

Name: 127.0.0.1:50010
Decommission Status : Normal
Configured Capacity: 211474616320 (196.95 GB)
DFS Used: 6953209856 (6.48 GB)
Non DFS Used: 42119852032 (39.23 GB)
DFS Remaining: 162401554432(151.25 GB)
DFS Used%: 3.29%
DFS Remaining%: 76.79%
Last contact: Fri Aug 10 16:59:26 PDT 2012

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 4:52 PM, Mohammad Tariq wrote:

> If your HDFS is in safemode you'll get something like this :
>
> cluster@ubuntu:~/hadoop-1.0.3$ bin/hadoop dfsadmin -report
> Safe mode is ON
> Configured Capacity: 31111143424 (28.97 GB)
> Present Capacity: 5309755392 (4.95 GB)
> DFS Remaining: 4799320064 (4.47 GB)
> DFS Used: 510435328 (486.79 MB)
> DFS Used%: 9.61%
> Under replicated blocks: 1
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 1 (1 total, 0 dead)
>
> Name: 127.0.0.1:50010
> Decommission Status : Normal
> Configured Capacity: 31111143424 (28.97 GB)
> DFS Used: 510435328 (486.79 MB)
> Non DFS Used: 25801388032 (24.03 GB)
> DFS Remaining: 4799320064(4.47 GB)
> DFS Used%: 1.64%
> DFS Remaining%: 15.43%
> Last contact: Sat Aug 11 05:19:18 IST 2012
>
> See the line in red color.
>
> Regards,
> Mohammad Tariq
>
>
> On Sat, Aug 11, 2012 at 5:20 AM, Mohammad Tariq <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
> > point your web browser to http://localhost:9000. It'll show all the
> > details of your HDFS.
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> > On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
> wrote:
> > > How do you check that?
> > >
> > > --
> > > Marco Gallotta | Mountain View, California
> > > Software Engineer, Infrastructure | Loki Studios
> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> > >
> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > >
> > >
> > > On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
> > >
> > > > This is pretty strange. I mean everything seems to be in place, but we
> > > > are stuck. Please make a check once if your Hdfs is in safemode.
> > > >
> > > > Regards,
> > > > Mohammad Tariq
> > > >
> > > >
> > > > On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))> wrote:
> > > > > What about fs.default.name (http://fs.default.name)?????
> > > > >
> > > > > Regards,
> > > > > Mohammad Tariq
> > > > >
> > > > >
> > > > > On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))> wrote:
> > > > > > It's in /var which is persistent across reboots.
> > > > > >
> > > > > > --
> > > > > > Marco Gallotta | Mountain View, California
> > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > >
> > > > >
> > > >
> > >
> >
>
> twitter.com/marcog (http://twitter.com/marcog)
> > > > > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650)
> > > > >
> > > >

Can you try to reboot the machine and run repair again. Might not sound logical but I would give it a shot.

PS: My personal experience is that hbase and hadoop has never been reliable in my standalone environment. I always trust the distributed cluster environment. AFAIK, these things are tested extensively in distributed mode.

Best Regards,
Anil

On Aug 10, 2012, at 5:00 PM, Marco Gallotta <[EMAIL PROTECTED]> wrote:

> Nope, not in safe mode. Gar, this is going nowhere. :/ Thanks for the help so far though!
>
> Configured Capacity: 211474616320 (196.95 GB)
> Present Capacity: 169354764288 (157.72 GB)
> DFS Remaining: 162401554432 (151.25 GB)
> DFS Used: 6953209856 (6.48 GB)
> DFS Used%: 4.11%
> Under replicated blocks: 436
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 1 (1 total, 0 dead)
>
> Name: 127.0.0.1:50010
> Decommission Status : Normal
> Configured Capacity: 211474616320 (196.95 GB)
> DFS Used: 6953209856 (6.48 GB)
> Non DFS Used: 42119852032 (39.23 GB)
> DFS Remaining: 162401554432(151.25 GB)
> DFS Used%: 3.29%
> DFS Remaining%: 76.79%
> Last contact: Fri Aug 10 16:59:26 PDT 2012
>
>
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> [EMAIL PROTECTED] | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 4:52 PM, Mohammad Tariq wrote:
>
>> If your HDFS is in safemode you'll get something like this :
>>
>> cluster@ubuntu:~/hadoop-1.0.3$ bin/hadoop dfsadmin -report
>> Safe mode is ON
>> Configured Capacity: 31111143424 (28.97 GB)
>> Present Capacity: 5309755392 (4.95 GB)
>> DFS Remaining: 4799320064 (4.47 GB)
>> DFS Used: 510435328 (486.79 MB)
>> DFS Used%: 9.61%
>> Under replicated blocks: 1
>> Blocks with corrupt replicas: 0
>> Missing blocks: 0
>>
>> -------------------------------------------------
>> Datanodes available: 1 (1 total, 0 dead)
>>
>> Name: 127.0.0.1:50010
>> Decommission Status : Normal
>> Configured Capacity: 31111143424 (28.97 GB)
>> DFS Used: 510435328 (486.79 MB)
>> Non DFS Used: 25801388032 (24.03 GB)
>> DFS Remaining: 4799320064(4.47 GB)
>> DFS Used%: 1.64%
>> DFS Remaining%: 15.43%
>> Last contact: Sat Aug 11 05:19:18 IST 2012
>>
>> See the line in red color.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>> On Sat, Aug 11, 2012 at 5:20 AM, Mohammad Tariq <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
>>> You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
>>> point your web browser to http://localhost:9000. It'll show all the
>>> details of your HDFS.
>>>
>>> Regards,
>>> Mohammad Tariq
>>>
>>>
>>> On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
>> wrote:
>>>> How do you check that?
>>>>
>>>> --
>>>> Marco Gallotta | Mountain View, California
>>>> Software Engineer, Infrastructure | Loki Studios
>>>> fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
>>>> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
>>>>
>>>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>>>
>>>>
>>>> On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
>>>>
>>>>> This is pretty strange. I mean everything seems to be in place, but we
>>>>> are stuck. Please make a check once if your Hdfs is in safemode.
>>>>>
>>>>> Regards,
>>>>> Mohammad Tariq
>>>>>
>>>>>
>>>>> On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
>> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))> wrote:
>>>>>> What about fs.default.name (http://fs.default.name)?????
>>>>>>
>>>>>> Regards,
>>>>>> Mohammad Tariq
>>>>>>
>>>>>>
>>>>>> On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
>> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))> wrote:

Same exception on repair after reboot. :/

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 5:09 PM, Anil Gupta wrote:

> Can you try to reboot the machine and run repair again. Might not sound logical but I would give it a shot.
>
> PS: My personal experience is that hbase and hadoop has never been reliable in my standalone environment. I always trust the distributed cluster environment. AFAIK, these things are tested extensively in distributed mode.
>
> Best Regards,
> Anil
>
> On Aug 10, 2012, at 5:00 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
>
> > Nope, not in safe mode. Gar, this is going nowhere. :/ Thanks for the help so far though!
> >
> > Configured Capacity: 211474616320 (196.95 GB)
> > Present Capacity: 169354764288 (157.72 GB)
> > DFS Remaining: 162401554432 (151.25 GB)
> > DFS Used: 6953209856 (6.48 GB)
> > DFS Used%: 4.11%
> > Under replicated blocks: 436
> > Blocks with corrupt replicas: 0
> > Missing blocks: 0
> >
> > -------------------------------------------------
> > Datanodes available: 1 (1 total, 0 dead)
> >
> > Name: 127.0.0.1:50010
> > Decommission Status : Normal
> > Configured Capacity: 211474616320 (196.95 GB)
> > DFS Used: 6953209856 (6.48 GB)
> > Non DFS Used: 42119852032 (39.23 GB)
> > DFS Remaining: 162401554432(151.25 GB)
> > DFS Used%: 3.29%
> > DFS Remaining%: 76.79%
> > Last contact: Fri Aug 10 16:59:26 PDT 2012
> >
> >
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > On Friday 10 August 2012 at 4:52 PM, Mohammad Tariq wrote:
> >
> > > If your HDFS is in safemode you'll get something like this :
> > >
> > > cluster@ubuntu:~/hadoop-1.0.3$ bin/hadoop dfsadmin -report
> > > Safe mode is ON
> > > Configured Capacity: 31111143424 (28.97 GB)
> > > Present Capacity: 5309755392 (4.95 GB)
> > > DFS Remaining: 4799320064 (4.47 GB)
> > > DFS Used: 510435328 (486.79 MB)
> > > DFS Used%: 9.61%
> > > Under replicated blocks: 1
> > > Blocks with corrupt replicas: 0
> > > Missing blocks: 0
> > >
> > > -------------------------------------------------
> > > Datanodes available: 1 (1 total, 0 dead)
> > >
> > > Name: 127.0.0.1:50010
> > > Decommission Status : Normal
> > > Configured Capacity: 31111143424 (28.97 GB)
> > > DFS Used: 510435328 (486.79 MB)
> > > Non DFS Used: 25801388032 (24.03 GB)
> > > DFS Remaining: 4799320064(4.47 GB)
> > > DFS Used%: 1.64%
> > > DFS Remaining%: 15.43%
> > > Last contact: Sat Aug 11 05:19:18 IST 2012
> > >
> > > See the line in red color.
> > >
> > > Regards,
> > > Mohammad Tariq
> > >
> > >
> > > On Sat, Aug 11, 2012 at 5:20 AM, Mohammad Tariq <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > > You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
> > > > point your web browser to http://localhost:9000. It'll show all the
> > > > details of your HDFS.
> > > >
> > > > Regards,
> > > > Mohammad Tariq
> > > >
> > > >
> > > > On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
> > > wrote:
> > > > > How do you check that?
> > > > >
> > > > > --
> > > > > Marco Gallotta | Mountain View, California
> > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > > > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> > > > >
> > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > >

Yeah, I feel the same. I am still in the learning phase, so it is quite
possible that I might be missing something important. But we have several
experts on the list and I hope they post their response for you.

Regards,
    Mohammad Tariq

On Sat, Aug 11, 2012 at 5:30 AM, Marco Gallotta <[EMAIL PROTECTED]>wrote:

> Nope, not in safe mode. Gar, this is going nowhere. :/ Thanks for the help
> so far though!
>
> Configured Capacity: 211474616320 (196.95 GB)
> Present Capacity: 169354764288 (157.72 GB)
> DFS Remaining: 162401554432 (151.25 GB)
> DFS Used: 6953209856 (6.48 GB)
> DFS Used%: 4.11%
> Under replicated blocks: 436
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 1 (1 total, 0 dead)
>
> Name: 127.0.0.1:50010
> Decommission Status : Normal
> Configured Capacity: 211474616320 (196.95 GB)
> DFS Used: 6953209856 (6.48 GB)
> Non DFS Used: 42119852032 (39.23 GB)
> DFS Remaining: 162401554432(151.25 GB)
> DFS Used%: 3.29%
> DFS Remaining%: 76.79%
> Last contact: Fri Aug 10 16:59:26 PDT 2012
>
>
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> [EMAIL PROTECTED] | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 4:52 PM, Mohammad Tariq wrote:
>
> > If your HDFS is in safemode you'll get something like this :
> >
> > cluster@ubuntu:~/hadoop-1.0.3$ bin/hadoop dfsadmin -report
> > Safe mode is ON
> > Configured Capacity: 31111143424 (28.97 GB)
> > Present Capacity: 5309755392 (4.95 GB)
> > DFS Remaining: 4799320064 (4.47 GB)
> > DFS Used: 510435328 (486.79 MB)
> > DFS Used%: 9.61%
> > Under replicated blocks: 1
> > Blocks with corrupt replicas: 0
> > Missing blocks: 0
> >
> > -------------------------------------------------
> > Datanodes available: 1 (1 total, 0 dead)
> >
> > Name: 127.0.0.1:50010
> > Decommission Status : Normal
> > Configured Capacity: 31111143424 (28.97 GB)
> > DFS Used: 510435328 (486.79 MB)
> > Non DFS Used: 25801388032 (24.03 GB)
> > DFS Remaining: 4799320064(4.47 GB)
> > DFS Used%: 1.64%
> > DFS Remaining%: 15.43%
> > Last contact: Sat Aug 11 05:19:18 IST 2012
> >
> > See the line in red color.
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> > On Sat, Aug 11, 2012 at 5:20 AM, Mohammad Tariq <[EMAIL PROTECTED](mailto:
> [EMAIL PROTECTED])> wrote:
> > > You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
> > > point your web browser to http://localhost:9000. It'll show all the
> > > details of your HDFS.
> > >
> > > Regards,
> > > Mohammad Tariq
> > >
> > >
> > > On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <[EMAIL PROTECTED](mailto:
> [EMAIL PROTECTED])>
> > wrote:
> > > > How do you check that?
> > > >
> > > > --
> > > > Marco Gallotta | Mountain View, California
> > > > Software Engineer, Infrastructure | Loki Studios
> > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> twitter.com/marcog (http://twitter.com/marcog)
> > > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650)
> 417-3313
> > > >
> > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > >
> > > >
> > > > On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
> > > >
> > > > > This is pretty strange. I mean everything seems to be in place,
> but we
> > > > > are stuck. Please make a check once if your Hdfs is in safemode.
> > > > >
> > > > > Regards,
> > > > > Mohammad Tariq
> > > > >
> > > > >
> > > > > On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <
> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))> wrote:
> > > > > > What about fs.default.name (http://fs.default.name)?????
> > > > > >
> > > > > > Regards,
> > > > > > Mohammad Tariq
> > > > > >
> > > > > >
> > > > > > On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <
> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto

Have you specified "hadoop.tmp.dir" property in your core-site.xml and
"dfs.data.dir" and "dfs.name.dir" properties in your hdfs-site.xml
files??
If not you will loose all your data along with you meta information as
Anil has said.

Regards,
    Mohammad Tariq
On Sat, Aug 11, 2012 at 5:01 AM, anil gupta <[EMAIL PROTECTED]> wrote:
> Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
> rebooted your machined then you will have problems.
>
> On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <[EMAIL PROTECTED]>wrote:
>
>> It's a pseudo-distributed cluster, as I plan to add more nodes as we start
>> gathering more data.
>>
>> I get the following error when running hbck -repair, and then it stalls:
>>
>> 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
>> after fix...
>> Version: 0.94.0
>> 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
>> 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
>> HDFS...
>> Exception in thread "main" java.util.concurrent.RejectedExecutionException
>> at
>> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
>> at
>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>> at
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>> at
>> org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
>> at
>> org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
>> at
>> org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
>> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
>> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
>>
>>
>>
>> --
>> Marco Gallotta | Mountain View, California
>> Software Engineer, Infrastructure | Loki Studios
>> fb.me/marco.gallotta | twitter.com/marcog
>> [EMAIL PROTECTED] | +1 (650) 417-3313
>>
>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>
>>
>> On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>>
>> > Is it a standalone installation or pseudo-distributed?
>> > I faced a similar problem a few days back in a distributed cluster and
>> used
>> > hbck -repair option. You might give it a try.
>> >
>> > ~Anil
>> >
>> > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <[EMAIL PROTECTED](mailto:
>> [EMAIL PROTECTED])> wrote:
>> >
>> > > Could you please share your /etc/hosts file??Meantime, do a manual
>> > > compaction and see if ti works.
>> > >
>> > > Regards,
>> > > Mohammad Tariq
>> > >
>> > >
>> > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <[EMAIL PROTECTED](mailto:
>> [EMAIL PROTECTED])>
>> > > wrote:
>> > > > It's not a distributed cluster. I'm not processing enough data yet.
>> So
>> > >
>> > > the reference to localhost is correct.
>> > > >
>> > > > --
>> > > > Marco Gallotta | Mountain View, California
>> > > > Software Engineer, Infrastructure | Loki Studios
>> > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>> twitter.com/marcog (http://twitter.com/marcog)
>> > > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650)
>> 417-3313
>> > > >
>> > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > > >
>> > > >
>> > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
>> > > >
>> > > > > Are you running a distributed cluster?
>> > > > > If yes, do you have localhost in /etc/hosts file?
>> > > > >
>> > > > > You are getting reference to localhost in hbck output:
>> > > > > ERROR: Region { meta => null, hdfs =>
>> > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
>> > > > > deployed => } on HDFS, but not listed in META or deployed on any
>> region
>> > > > > server
>> > > > >
>> > > > > ~Anil
>> > > > >
>> > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
>> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
>> > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))>wrote:

In core-site.xml:

  <property>
    <name>hadoop.tmp.dir</name>
    <value>/var/hadoop/${user.name}</value>
  </property>
In hdfs-site.xml:

  <property>
    <name>dfs.name.dir</name>
    <value>/var/hdfs-data-name</value>
  </property>
  <property>
    <name>dfs.data.dir</name>
    <value>/var/hdfs-data</value>
  </property>
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 4:34 PM, Mohammad Tariq wrote:

> Have you specified "hadoop.tmp.dir" property in your core-site.xml and
> "dfs.data.dir" and "dfs.name.dir" properties in your hdfs-site.xml
> files??
> If not you will loose all your data along with you meta information as
> Anil has said.
>
> Regards,
> Mohammad Tariq
>
>
> On Sat, Aug 11, 2012 at 5:01 AM, anil gupta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
> > rebooted your machined then you will have problems.
> >
> > On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>wrote:
> >
> > > It's a pseudo-distributed cluster, as I plan to add more nodes as we start
> > > gathering more data.
> > >
> > > I get the following error when running hbck -repair, and then it stalls:
> > >
> > > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
> > > after fix...
> > > Version: 0.94.0
> > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
> > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
> > > HDFS...
> > > Exception in thread "main" java.util.concurrent.RejectedExecutionException
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
> > > at
> > > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
> > > at
> > > org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
> > > at
> > > org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
> > > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
> > > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
> > >
> > >
> > >
> > > --
> > > Marco Gallotta | Mountain View, California
> > > Software Engineer, Infrastructure | Loki Studios
> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> > >
> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > >
> > >
> > > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
> > >
> > > > Is it a standalone installation or pseudo-distributed?
> > > > I faced a similar problem a few days back in a distributed cluster and
> > > >
> > >
> > > used
> > > > hbck -repair option. You might give it a try.
> > > >
> > > > ~Anil
> > > >
> > > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
> > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))> wrote:
> > > >
> > > > > Could you please share your /etc/hosts file??Meantime, do a manual
> > > > > compaction and see if ti works.
> > > > >
> > > > > Regards,
> > > > > Mohammad Tariq
> > > > >
> > > > >
> > > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
> > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))>
> > > > > wrote:
> > > > > > It's not a distributed cluster. I'm not processing enough data yet.
> > > > >
> > > > >
> > > >
> > >
> > > So

Nothing perculiar in /etc/hosts:

127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
Ran a major compaction with

echo 'list' | ./hbase shell | tail -n52 | head -n50 | while read t; do echo "major_compact '$t'"; done | ./hbase shell

And still get the 6 inconsistencies.

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 3:39 PM, Mohammad Tariq wrote:

> Could you please share your /etc/hosts file??Meantime, do a manual
> compaction and see if ti works.
>
> Regards,
> Mohammad Tariq
>
>
> On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > It's not a distributed cluster. I'm not processing enough data yet. So the reference to localhost is correct.
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> >
> > > Are you running a distributed cluster?
> > > If yes, do you have localhost in /etc/hosts file?
> > >
> > > You are getting reference to localhost in hbck output:
> > > ERROR: Region { meta => null, hdfs =>
> > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > deployed => } on HDFS, but not listed in META or deployed on any region
> > > server
> > >
> > > ~Anil
> > >
> > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>wrote:
> > >
> > > > Here's the output from hbck -details: http://pastebin.com/ZxVZEctY
> > > >
> > > > Extract:
> > > >
> > > > 6 inconsistencies detected.
> > > > Status: INCONSISTENT
> > > >
> > > > 6 is the number of tables that appear in "list" but cannot be operated on
> > > > (which btw, includes not being able to run disable/drop on them - both ops
> > > > say table not found). I also just noticed "foo" does not occur in a table
> > > > list, although I did create it at one point but was able to clear it from
> > > > .META. when it also was reporting table not found when trying to
> > > > disable/drop it. All these come from when I ^C'ed (i.e. killed) table
> > > > creation when I was trying to get lzo compression working and table
> > > > creation was hanging.
> > > >
> > > > Is there any way to repair this? I see hbck has repair options, but I want
> > > > to proceed with caution.
> > > >
> > > > --
> > > > Marco Gallotta | Mountain View, California
> > > > Software Engineer, Infrastructure | Loki Studios
> > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> > > >
> > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > >
> > > >
> > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > > >
> > > > > Hi Marco,
> > > > >
> > > > > Did anything disastrous happen to cluster?
> > > > > Can you try using hbck utility of HBase.
> > > > > Run: 'hbase hbck -help' to get all the available options.
> > > > >
> > > > > ~Anil
> > > > >
> > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])(mailto:
> > > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))>wrote:
> > > > >
> > > > > > Hi there
> > > > > >
> > > > > > I have a few tables which show up in a "list" in the shell, but produce
> > > > > > "table not found" when performing any operation on them. There is no

@Anil : Good Point.
@Marco : First make sure that all the AMIs running region servers are
reachable and there is no problem in DNS resolution.(As I see you are
using AWS).

Regards,
    Mohammad Tariq
On Sat, Aug 11, 2012 at 4:00 AM, anil gupta <[EMAIL PROTECTED]> wrote:
> Are you running a distributed cluster?
> If yes, do you have localhost in /etc/hosts file?
>
> You are getting reference to localhost in hbck output:
> ERROR: Region { meta => null, hdfs =>
> hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> deployed =>  } on HDFS, but not listed in META or deployed on any region
> server
>
> ~Anil
>
> On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <[EMAIL PROTECTED]>wrote:
>
>> Here's the output from hbck -details: http://pastebin.com/ZxVZEctY
>>
>> Extract:
>>
>> 6 inconsistencies detected.
>> Status: INCONSISTENT
>>
>> 6 is the number of tables that appear in "list" but cannot be operated on
>> (which btw, includes not being able to run disable/drop on them - both ops
>> say table not found). I also just noticed "foo" does not occur in a table
>> list, although I did create it at one point but was able to clear it from
>> .META. when it also was reporting table not found when trying to
>> disable/drop it. All these come from when I ^C'ed (i.e. killed) table
>> creation when I was trying to get lzo compression working and table
>> creation was hanging.
>>
>> Is there any way to repair this? I see hbck has repair options, but I want
>> to proceed with caution.
>>
>> --
>> Marco Gallotta | Mountain View, California
>> Software Engineer, Infrastructure | Loki Studios
>> fb.me/marco.gallotta | twitter.com/marcog
>> [EMAIL PROTECTED] | +1 (650) 417-3313
>>
>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>
>>
>> On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>>
>> > Hi Marco,
>> >
>> > Did anything disastrous happen to cluster?
>> > Can you try using hbck utility of HBase.
>> > Run: 'hbase hbck -help' to get all the available options.
>> >
>> > ~Anil
>> >
>> > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <[EMAIL PROTECTED](mailto:
>> [EMAIL PROTECTED])>wrote:
>> >
>> > > Hi there
>> > >
>> > > I have a few tables which show up in a "list" in the shell, but produce
>> > > "table not found" when performing any operation on them. There is no
>> > > reference of them in the .META. table. It seems to be resulting in
>> some of
>> > > the hbase services being killed every so often.
>> > >
>> > > Here are some logs from master (foo is one of the tables not found):
>> > >
>> > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
>> > > Master server abort: loaded coprocessors are: []
>> > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
>> > > Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>> > > state=PENDING_OPEN, ts=1344570044277,
>> > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>> ..
>> > > Cannot transit it to OFFLINE.
>> > >
>> > >
>> > > There are also a number of the following types of error logs:
>> > >
>> > > 2012-08-09 20:10:04,308 ERROR
>> > > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
>> > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
>> > >
>> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
>> > > Received:OPEN for the
>> > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we
>> are
>> > > already trying to OPEN.
>> > >
>> > > Any ideas how to find and remove any references to these non-existent
>> > > tables?
>> > >
>> > > --
>> > > Marco Gallotta | Mountain View, California
>> > > Software Engineer, Infrastructure | Loki Studios
>> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>> twitter.com/marcog (http://twitter.com/marcog)
>> > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
>> > >
>> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

Hi Marco,

      There seems to be some problem with the regionserver hosting the
table. Had you disabled or deleted "foo"?And try to see what "hbck"
says. And RegionAlreadyInTransitionException: is normally thrown if a
region server is asked to open or close a region but it's already
processing that region. BTW, did you find anything abnormal with your
HDFS??
Regards,
    Mohammad Tariq
On Sat, Aug 11, 2012 at 2:52 AM, Marco Gallotta <[EMAIL PROTECTED]> wrote:
> Hi there
>
> I have a few tables which show up in a "list" in the shell, but produce "table not found" when performing any operation on them. There is no reference of them in the .META. table. It seems to be resulting in some of the hbase services being killed every so often.
>
> Here are some logs from master (foo is one of the tables not found):
>
> 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
> 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. state=PENDING_OPEN, ts=1344570044277, server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 .. Cannot transit it to OFFLINE.
>
>
> There are also a number of the following types of error logs:
>
> 2012-08-09 20:10:04,308 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are already trying to OPEN.
>
> Any ideas how to find and remove any references to these non-existent tables?
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> [EMAIL PROTECTED] | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>

Did you blow away your cluster once (in the sense of the deleting all data and starting from scratch)?This could be left-over data in ZK.

----- Original Message -----
From: Marco Gallotta <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Cc:
Sent: Friday, August 10, 2012 2:22 PM
Subject: Table listed in "list", but not in .META.

Hi there

I have a few tables which show up in a "list" in the shell, but produce "table not found" when performing any operation on them. There is no reference of them in the .META. table. It seems to be resulting in some of the hbase services being killed every so often.

Here are some logs from master (foo is one of the tables not found):

2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. state=PENDING_OPEN, ts=1344570044277, server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 .. Cannot transit it to OFFLINE.
There are also a number of the following types of error logs:

2012-08-09 20:10:04,308 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are already trying to OPEN.

Any ideas how to find and remove any references to these non-existent tables?

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

I never did that, but I mentioned above that all these come from when I ^C'ed (i.e. killed) table creation when I was trying to get lzo compression working and table creation was hanging. Where would I check for references to these tables in ZK?

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 5:15 PM, lars hofhansl wrote:

> Did you blow away your cluster once (in the sense of the deleting all data and starting from scratch)?This could be left-over data in ZK.
>
>
>
> ----- Original Message -----
> From: Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
> To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> Cc:
> Sent: Friday, August 10, 2012 2:22 PM
> Subject: Table listed in "list", but not in .META.
>
> Hi there
>
> I have a few tables which show up in a "list" in the shell, but produce "table not found" when performing any operation on them. There is no reference of them in the .META. table. It seems to be resulting in some of the hbase services being killed every so often.
>
> Here are some logs from master (foo is one of the tables not found):
>
> 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
> 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. state=PENDING_OPEN, ts=1344570044277, server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 .. Cannot transit it to OFFLINE.
>
>
> There are also a number of the following types of error logs:
>
> 2012-08-09 20:10:04,308 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are already trying to OPEN.
>
> Any ideas how to find and remove any references to these non-existent tables?
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

w00t! I looked in "/hbase/" on the hdfs and found directories referencing the stale tables and deleting them worked. It now says 0 inconsistencies. Maybe worth adding this to the hbck -repair script? Anyway, thanks everyone for the help! Hopefully this stops the sporadic crashes I was seeing.

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 5:20 PM, Marco Gallotta wrote:

> I never did that, but I mentioned above that all these come from when I ^C'ed (i.e. killed) table creation when I was trying to get lzo compression working and table creation was hanging. Where would I check for references to these tables in ZK?
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 5:15 PM, lars hofhansl wrote:
>
> > Did you blow away your cluster once (in the sense of the deleting all data and starting from scratch)?This could be left-over data in ZK.
> >
> >
> >
> > ----- Original Message -----
> > From: Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
> > To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> > Cc:
> > Sent: Friday, August 10, 2012 2:22 PM
> > Subject: Table listed in "list", but not in .META.
> >
> > Hi there
> >
> > I have a few tables which show up in a "list" in the shell, but produce "table not found" when performing any operation on them. There is no reference of them in the .META. table. It seems to be resulting in some of the hbase services being killed every so often.
> >
> > Here are some logs from master (foo is one of the tables not found):
> >
> > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
> > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. state=PENDING_OPEN, ts=1344570044277, server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 .. Cannot transit it to OFFLINE.
> >
> >
> > There are also a number of the following types of error logs:
> >
> > 2012-08-09 20:10:04,308 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are already trying to OPEN.
> >
> > Any ideas how to find and remove any references to these non-existent tables?
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>

Good to hear that... :) I also had a very bad crash of HBase a few days
ago... at that time had to run hbck repair to fix some errors and to fix
others i had to remove files from /hbase dir. It's a fairly complex system
so sometimes it really hard to nail the problem.

~Anil

On Fri, Aug 10, 2012 at 8:21 PM, Marco Gallotta <[EMAIL PROTECTED]>wrote:

> w00t! I looked in "/hbase/" on the hdfs and found directories referencing
> the stale tables and deleting them worked. It now says 0 inconsistencies.
> Maybe worth adding this to the hbck -repair script? Anyway, thanks everyone
> for the help! Hopefully this stops the sporadic crashes I was seeing.
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> [EMAIL PROTECTED] | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 5:20 PM, Marco Gallotta wrote:
>
> > I never did that, but I mentioned above that all these come from when I
> ^C'ed (i.e. killed) table creation when I was trying to get lzo compression
> working and table creation was hanging. Where would I check for references
> to these tables in ZK?
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog(
> http://twitter.com/marcog)
> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > On Friday 10 August 2012 at 5:15 PM, lars hofhansl wrote:
> >
> > > Did you blow away your cluster once (in the sense of the deleting all
> data and starting from scratch)?This could be left-over data in ZK.
> > >
> > >
> > >
> > > ----- Original Message -----
> > > From: Marco Gallotta <[EMAIL PROTECTED] (mailto:
> [EMAIL PROTECTED])>
> > > To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> > > Cc:
> > > Sent: Friday, August 10, 2012 2:22 PM
> > > Subject: Table listed in "list", but not in .META.
> > >
> > > Hi there
> > >
> > > I have a few tables which show up in a "list" in the shell, but
> produce "table not found" when performing any operation on them. There is
> no reference of them in the .META. table. It seems to be resulting in some
> of the hbase services being killed every so often.
> > >
> > > Here are some logs from master (foo is one of the tables not found):
> > >
> > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> Master server abort: loaded coprocessors are: []
> > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> state=PENDING_OPEN, ts=1344570044277,
> server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 ..
> Cannot transit it to OFFLINE.
> > >
> > >
> > > There are also a number of the following types of error logs:
> > >
> > > 2012-08-09 20:10:04,308 ERROR
> org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
> ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> Received:OPEN for the
> region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are
> already trying to OPEN.
> > >
> > > Any ideas how to find and remove any references to these non-existent
> tables?
> > >
> > > --
> > > Marco Gallotta | Mountain View, California
> > > Software Engineer, Infrastructure | Loki Studios
> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> twitter.com/marcog (http://twitter.com/marcog)
> > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
> > >
> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
>
>
--
Thanks & Regards,
Anil Gupta