Hi all,

i have now a big problem with one region. This region is allways in
transition and i don't no how fix it.

I run "hbase hbck -repair" and this ended with:
INFO util.HBaseFsckRepair: Region still in transition, waiting for it
to become assigned: {NAME =>
'twitter_tweets,08000e806a7b8ba7d6cdd5c9c0956e7e,1350294699032.3a932a1e2b15abbf7539ca8ad761ecfc.',
STARTKEY => '08000e806a7b8ba7d6cdd5c9c0956e7e', ENDKEY =>
'1002e24a61f8ea1ce10267e09e377aa5', ENCODED =>
3a932a1e2b15abbf7539ca8ad761ecfc,

How i can bring it back online?

Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3

I found this in master log http://pastebin.com/CYt5PZCL and this line
are repeated for all region servers.

Can someone help me please?

Have you tried clearing out your Znode information?  Typically, when I have
encountered a RIT, we will bring down HBase and go to the ZKcli and clear
out /hbase.  What do you see in the logs pertaining to the region?  If it
is a region that has bad hfiles or something like that you will not be able
to easily get rid of it through clearing out the ZK.

On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <[EMAIL PROTECTED]> wrote:

> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
>
> I found this in master log http://pastebin.com/CYt5PZCL and this line
> are repeated for all region servers.
>
> Can someone help me please?
>

--
Kevin O'Dell
Customer Operations Engineer, Cloudera

Hello Kevin,

thanks for response. I clear it now and start cluster, but now i have
many other's regions :(
Here is output from hbase hbck -fixmeta
http://pastebin.com/HqsPVLMi

Any next hint?

Thanks a lot

2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
> Have you tried clearing out your Znode information?  Typically, when I have
> encountered a RIT, we will bring down HBase and go to the ZKcli and clear
> out /hbase.  What do you see in the logs pertaining to the region?  If it
> is a region that has bad hfiles or something like that you will not be able
> to easily get rid of it through clearing out the ZK.
>
> On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <[EMAIL PROTECTED]> wrote:
>
>> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
>>
>> I found this in master log http://pastebin.com/CYt5PZCL and this line
>> are repeated for all region servers.
>>
>> Can someone help me please?
>>
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera

--
Save The World - http://www.worldcommunitygrid.org/
http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR

LesTR

Lukas,

  Sure, how long has it been since the restart?  You will need to give the
regions time to transition and logs time to split.  Did the region
in transition properly?  Can you please out a pastebin together of hbck
-details full output so that I can take a look at it.  Once I have reviewed
it, I should be able to provide you with a plan to out this back together.

On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <[EMAIL PROTECTED]> wrote:

> Hello Kevin,
>
> thanks for response. I clear it now and start cluster, but now i have
> many other's regions :(
> Here is output from hbase hbck -fixmeta
> http://pastebin.com/HqsPVLMi
>
> Any next hint?
>
> Thanks a lot
>
> 2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
> > Have you tried clearing out your Znode information?  Typically, when I
> have
> > encountered a RIT, we will bring down HBase and go to the ZKcli and clear
> > out /hbase.  What do you see in the logs pertaining to the region?  If it
> > is a region that has bad hfiles or something like that you will not be
> able
> > to easily get rid of it through clearing out the ZK.
> >
> > On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <[EMAIL PROTECTED]>
> wrote:
> >
> >> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
> >>
> >> I found this in master log http://pastebin.com/CYt5PZCL and this line
> >> are repeated for all region servers.
> >>
> >> Can someone help me please?
> >>
> >
> >
> >
> > --
> > Kevin O'Dell
> > Customer Operations Engineer, Cloudera
>
>
>
> --
> Save The World - http://www.worldcommunitygrid.org/
> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>
> LesTR
>

--
Kevin O'Dell
Customer Operations Engineer, Cloudera

In the past when I have seen regions get locked in transition it is usually
a problem with the HMaster.  Seemingly a transition starts and succeeds
between region servers, but the HMaster may miss part of that communication
and think it is still in transition.  Then it keeps retrying but the node
is not where it expects anymore.  This doesn't affect normal operations,
except that the balancer cannot run while something is in transition.
 Restarting the HMaster always fixed this for me and had no adverse affects
that I could see.

On Mon, Oct 15, 2012 at 10:28 AM, Kevin O'dell <[EMAIL PROTECTED]>wrote:

> Lukas,
>
>   Sure, how long has it been since the restart?  You will need to give the
> regions time to transition and logs time to split.  Did the region
> in transition properly?  Can you please out a pastebin together of hbck
> -details full output so that I can take a look at it.  Once I have reviewed
> it, I should be able to provide you with a plan to out this back together.
>
> On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <[EMAIL PROTECTED]>
> wrote:
>
> > Hello Kevin,
> >
> > thanks for response. I clear it now and start cluster, but now i have
> > many other's regions :(
> > Here is output from hbase hbck -fixmeta
> > http://pastebin.com/HqsPVLMi
> >
> > Any next hint?
> >
> > Thanks a lot
> >
> > 2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
> > > Have you tried clearing out your Znode information?  Typically, when I
> > have
> > > encountered a RIT, we will bring down HBase and go to the ZKcli and
> clear
> > > out /hbase.  What do you see in the logs pertaining to the region?  If
> it
> > > is a region that has bad hfiles or something like that you will not be
> > able
> > > to easily get rid of it through clearing out the ZK.
> > >
> > > On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <[EMAIL PROTECTED]>
> > wrote:
> > >
> > >> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
> > >>
> > >> I found this in master log http://pastebin.com/CYt5PZCL and this line
> > >> are repeated for all region servers.
> > >>
> > >> Can someone help me please?
> > >>
> > >
> > >
> > >
> > > --
> > > Kevin O'Dell
> > > Customer Operations Engineer, Cloudera
> >
> >
> >
> > --
> > Save The World - http://www.worldcommunitygrid.org/
> > http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
> >
> > LesTR
> >
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
>

Kevin,

Its more than one hour and region are still in transition.
Here is output from hbck -detailed http://pastebin.com/f9wLx9LU

Thanks for time

Lukas

2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
> Lukas,
>
>   Sure, how long has it been since the restart?  You will need to give the
> regions time to transition and logs time to split.  Did the region
> in transition properly?  Can you please out a pastebin together of hbck
> -details full output so that I can take a look at it.  Once I have reviewed
> it, I should be able to provide you with a plan to out this back together.
>
> On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <[EMAIL PROTECTED]> wrote:
>
>> Hello Kevin,
>>
>> thanks for response. I clear it now and start cluster, but now i have
>> many other's regions :(
>> Here is output from hbase hbck -fixmeta
>> http://pastebin.com/HqsPVLMi
>>
>> Any next hint?
>>
>> Thanks a lot
>>
>> 2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
>> > Have you tried clearing out your Znode information?  Typically, when I
>> have
>> > encountered a RIT, we will bring down HBase and go to the ZKcli and clear
>> > out /hbase.  What do you see in the logs pertaining to the region?  If it
>> > is a region that has bad hfiles or something like that you will not be
>> able
>> > to easily get rid of it through clearing out the ZK.
>> >
>> > On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
>> >>
>> >> I found this in master log http://pastebin.com/CYt5PZCL and this line
>> >> are repeated for all region servers.
>> >>
>> >> Can someone help me please?
>> >>
>> >
>> >
>> >
>> > --
>> > Kevin O'Dell
>> > Customer Operations Engineer, Cloudera
>>
>>
>>
>> --
>> Save The World - http://www.worldcommunitygrid.org/
>> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>>
>> LesTR
>>
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera

--
Save The World - http://www.worldcommunitygrid.org/
http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR

LesTR

Lukas,

  Looking over what you sent me:

ERROR: Region { meta =>
twitter_users,1ffd1a52913c3fd600dedb97d1b2b8ce,1350294703353.6e3be1c26233ed744479289b00a422dd.,
hdfs =>
hdfs://hadoop-7:9000/hbase/twitter_users/6e3be1c26233ed744479289b00a422dd,
deployed =>  } not deployed on any region server.
ERROR: Region { meta =>
facebook_posts,6103caf551a3115ee22e527542464a33,1350306570677.7112dd6fd61f2cf2675696c8320f7639.,
hdfs =>
hdfs://hadoop-7:9000/hbase/facebook_posts/7112dd6fd61f2cf2675696c8320f7639,
deployed =>  } not deployed on any region server.
ERROR: Region { meta =>
facebook_posts,81a92d771ace1b49526b0c940fc65fb5,1350294695237.8f290a471d0cd0d1036b306fa526db4e.,
hdfs =>
hdfs://hadoop-7:9000/hbase/facebook_posts/8f290a471d0cd0d1036b306fa526db4e,
deployed =>  } not deployed on any region server.
ERROR: Region { meta =>
facebook_posts,f01f68e30ffbeca7d248091f5319ffee,1350306575742.952fa87b758d1605bb263535376fdf33.,
hdfs =>
hdfs://hadoop-7:9000/hbase/facebook_posts/952fa87b758d1605bb263535376fdf33,
deployed =>  } not deployed on any region server.
ERROR: Region { meta =>
twitter_tweets,4fc3d2db81837d6aafb97348da86bcd7,1350308118451.a532656e5d112c2962648cd5598c635d.,
hdfs =>
hdfs://hadoop-7:9000/hbase/twitter_tweets/a532656e5d112c2962648cd5598c635d,
deployed =>  } not deployed on any region server.
ERROR: Region { meta =>
twitter_users,afcb98b707ee4ee8a830827e8a7265ed,1350294705684.b222aa685d8a53ba836518074b7e27a0.,
hdfs =>
hdfs://hadoop-7:9000/hbase/twitter_users/b222aa685d8a53ba836518074b7e27a0,
deployed =>  } not deployed on any region server.
ERROR: Region { meta =>
facebook_comments,e0405892a3d2065201252311ece8b5f9,1350306514155.c60bdbfcba96d4b381d7470756b15fdd.,
hdfs =>
hdfs://hadoop-7:9000/hbase/facebook_comments/c60bdbfcba96d4b381d7470756b15fdd,
deployed =>  } not deployed on any region server.
ERROR: Region { meta =>
facebook_posts,20600f7f12eb2d850c4e75d7e6dcbb84,1350294692505.d0192a38206a8df60210f6d972807f64.,
hdfs =>
hdfs://hadoop-7:9000/hbase/facebook_posts/d0192a38206a8df60210f6d972807f64,
deployed =>  } not deployed on any region server.

Does FSCK come back clean?  Are those the regions showing in transition?
 We are not going to be able to get a clear idea of what to do next until
we gather some more data.  At this point running repairs could put your
data in jeopardy.

On Mon, Oct 15, 2012 at 10:58 AM, Lukáš Drbal <[EMAIL PROTECTED]> wrote:

> Kevin,
>
> Its more than one hour and region are still in transition.
> Here is output from hbck -detailed http://pastebin.com/f9wLx9LU
>
> Thanks for time
>
> Lukas
>
> 2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
> > Lukas,
> >
> >   Sure, how long has it been since the restart?  You will need to give
> the
> > regions time to transition and logs time to split.  Did the region
> > in transition properly?  Can you please out a pastebin together of hbck
> > -details full output so that I can take a look at it.  Once I have
> reviewed
> > it, I should be able to provide you with a plan to out this back
> together.
> >
> > On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hello Kevin,
> >>
> >> thanks for response. I clear it now and start cluster, but now i have
> >> many other's regions :(
> >> Here is output from hbase hbck -fixmeta
> >> http://pastebin.com/HqsPVLMi
> >>
> >> Any next hint?
> >>
> >> Thanks a lot
> >>
> >> 2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
> >> > Have you tried clearing out your Znode information?  Typically, when I
> >> have
> >> > encountered a RIT, we will bring down HBase and go to the ZKcli and
> clear
> >> > out /hbase.  What do you see in the logs pertaining to the region?
>  If it
> >> > is a region that has bad hfiles or something like that you will not be
> >> able
> >> > to easily get rid of it through clearing out the ZK.
> >> >
> >> > On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <[EMAIL PROTECTED]>
> >> wrote:
> >> >
> >> >> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3

Kevin O'Dell
Customer Operations Engineer, Cloudera

Hi again.

> Does FSCK come back clean?  Are those the regions showing in transition?
>  We are not going to be able to get a clear idea of what to do next until
> we gather some more data.  At this point running repairs could put your
> data in jeopardy.
Ye, this is in transition, but from hadoop fsck are all ok and
consistent - reported only 14 undereplicated blocks from another
directory (not /hbase).

I must bring cluster back online, so i move /hbase on hdfs to
/hbase.old and start clear cluster with new tables :(

Can you give me please any suggestion how i can put data from old
files to new tables?

Thanks

My today's night will be very long :(

Lukas

>
> On Mon, Oct 15, 2012 at 10:58 AM, Lukáš Drbal <[EMAIL PROTECTED]> wrote:
>
>> Kevin,
>>
>> Its more than one hour and region are still in transition.
>> Here is output from hbck -detailed http://pastebin.com/f9wLx9LU
>>
>> Thanks for time
>>
>> Lukas
>>
>> 2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
>> > Lukas,
>> >
>> >   Sure, how long has it been since the restart?  You will need to give
>> the
>> > regions time to transition and logs time to split.  Did the region
>> > in transition properly?  Can you please out a pastebin together of hbck
>> > -details full output so that I can take a look at it.  Once I have
>> reviewed
>> > it, I should be able to provide you with a plan to out this back
>> together.
>> >
>> > On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >> Hello Kevin,
>> >>
>> >> thanks for response. I clear it now and start cluster, but now i have
>> >> many other's regions :(
>> >> Here is output from hbase hbck -fixmeta
>> >> http://pastebin.com/HqsPVLMi
>> >>
>> >> Any next hint?
>> >>
>> >> Thanks a lot
>> >>
>> >> 2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
>> >> > Have you tried clearing out your Znode information?  Typically, when I
>> >> have
>> >> > encountered a RIT, we will bring down HBase and go to the ZKcli and
>> clear
>> >> > out /hbase.  What do you see in the logs pertaining to the region?
>>  If it
>> >> > is a region that has bad hfiles or something like that you will not be
>> >> able
>> >> > to easily get rid of it through clearing out the ZK.
>> >> >
>> >> > On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <[EMAIL PROTECTED]>
>> >> wrote:
>> >> >
>> >> >> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
>> >> >>
>> >> >> I found this in master log http://pastebin.com/CYt5PZCL and this
>> line
>> >> >> are repeated for all region servers.
>> >> >>
>> >> >> Can someone help me please?
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Kevin O'Dell
>> >> > Customer Operations Engineer, Cloudera
>> >>
>> >>
>> >>
>> >> --
>> >> Save The World - http://www.worldcommunitygrid.org/
>> >> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>> >>
>> >> LesTR
>> >>
>> >
>> >
>> >
>> > --
>> > Kevin O'Dell
>> > Customer Operations Engineer, Cloudera
>>
>>
>>
>> --
>> Save The World - http://www.worldcommunitygrid.org/
>> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>>
>> LesTR
>>
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera

--
Save The World - http://www.worldcommunitygrid.org/
http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR

LesTR

Lukas,

  A little trick you can use is to just copy the table directories into
your new /hbase dir and then use:

hbck -fixMeta -fixAssignments

This will pull in the tables into your new empty META.  If once you do this
Regions get stuck in transition again, you have bigger problems.

On Mon, Oct 15, 2012 at 11:48 AM, Lukáš Drbal <[EMAIL PROTECTED]> wrote:

> Hi again.
>
> > Does FSCK come back clean?  Are those the regions showing in transition?
> >  We are not going to be able to get a clear idea of what to do next until
> > we gather some more data.  At this point running repairs could put your
> > data in jeopardy.
>
>
> Ye, this is in transition, but from hadoop fsck are all ok and
> consistent - reported only 14 undereplicated blocks from another
> directory (not /hbase).
>
> I must bring cluster back online, so i move /hbase on hdfs to
> /hbase.old and start clear cluster with new tables :(
>
> Can you give me please any suggestion how i can put data from old
> files to new tables?
>
> Thanks
>
> My today's night will be very long :(
>
> Lukas
>
> >
> > On Mon, Oct 15, 2012 at 10:58 AM, Lukáš Drbal <[EMAIL PROTECTED]>
> wrote:
> >
> >> Kevin,
> >>
> >> Its more than one hour and region are still in transition.
> >> Here is output from hbck -detailed http://pastebin.com/f9wLx9LU
> >>
> >> Thanks for time
> >>
> >> Lukas
> >>
> >> 2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
> >> > Lukas,
> >> >
> >> >   Sure, how long has it been since the restart?  You will need to give
> >> the
> >> > regions time to transition and logs time to split.  Did the region
> >> > in transition properly?  Can you please out a pastebin together of
> hbck
> >> > -details full output so that I can take a look at it.  Once I have
> >> reviewed
> >> > it, I should be able to provide you with a plan to out this back
> >> together.
> >> >
> >> > On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <[EMAIL PROTECTED]>
> >> wrote:
> >> >
> >> >> Hello Kevin,
> >> >>
> >> >> thanks for response. I clear it now and start cluster, but now i have
> >> >> many other's regions :(
> >> >> Here is output from hbase hbck -fixmeta
> >> >> http://pastebin.com/HqsPVLMi
> >> >>
> >> >> Any next hint?
> >> >>
> >> >> Thanks a lot
> >> >>
> >> >> 2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
> >> >> > Have you tried clearing out your Znode information?  Typically,
> when I
> >> >> have
> >> >> > encountered a RIT, we will bring down HBase and go to the ZKcli and
> >> clear
> >> >> > out /hbase.  What do you see in the logs pertaining to the region?
> >>  If it
> >> >> > is a region that has bad hfiles or something like that you will
> not be
> >> >> able
> >> >> > to easily get rid of it through clearing out the ZK.
> >> >> >
> >> >> > On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <
> [EMAIL PROTECTED]>
> >> >> wrote:
> >> >> >
> >> >> >> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
> >> >> >>
> >> >> >> I found this in master log http://pastebin.com/CYt5PZCL and this
> >> line
> >> >> >> are repeated for all region servers.
> >> >> >>
> >> >> >> Can someone help me please?
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Kevin O'Dell
> >> >> > Customer Operations Engineer, Cloudera
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Save The World - http://www.worldcommunitygrid.org/
> >> >>
> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
> >> >>
> >> >> LesTR
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Kevin O'Dell
> >> > Customer Operations Engineer, Cloudera
> >>
> >>
> >>
> >> --
> >> Save The World - http://www.worldcommunitygrid.org/
> >> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
> >>
> >> LesTR
> >>
> >
> >
> >
> > --
> > Kevin O'Dell
> > Customer Operations Engineer, Cloudera
>
>
>
> --
> Save The World - http://www.worldcommunitygrid.org/
> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>
> LesTR
>

--
Kevin O'Dell
Customer Operations Engineer, Cloudera

Wow, its perfect "hack" but what about "exists" rows?
For example when i have saved new version from row in /hbase i can't
replace it from /hbase.old

Thanks for time

Lukas

2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
> Lukas,
>
>   A little trick you can use is to just copy the table directories into
> your new /hbase dir and then use:
>
> hbck -fixMeta -fixAssignments
>
> This will pull in the tables into your new empty META.  If once you do this
> Regions get stuck in transition again, you have bigger problems.
>
> On Mon, Oct 15, 2012 at 11:48 AM, Lukáš Drbal <[EMAIL PROTECTED]> wrote:
>
>> Hi again.
>>
>> > Does FSCK come back clean?  Are those the regions showing in transition?
>> >  We are not going to be able to get a clear idea of what to do next until
>> > we gather some more data.  At this point running repairs could put your
>> > data in jeopardy.
>>
>>
>> Ye, this is in transition, but from hadoop fsck are all ok and
>> consistent - reported only 14 undereplicated blocks from another
>> directory (not /hbase).
>>
>> I must bring cluster back online, so i move /hbase on hdfs to
>> /hbase.old and start clear cluster with new tables :(
>>
>> Can you give me please any suggestion how i can put data from old
>> files to new tables?
>>
>> Thanks
>>
>> My today's night will be very long :(
>>
>> Lukas
>>
>> >
>> > On Mon, Oct 15, 2012 at 10:58 AM, Lukáš Drbal <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >> Kevin,
>> >>
>> >> Its more than one hour and region are still in transition.
>> >> Here is output from hbck -detailed http://pastebin.com/f9wLx9LU
>> >>
>> >> Thanks for time
>> >>
>> >> Lukas
>> >>
>> >> 2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
>> >> > Lukas,
>> >> >
>> >> >   Sure, how long has it been since the restart?  You will need to give
>> >> the
>> >> > regions time to transition and logs time to split.  Did the region
>> >> > in transition properly?  Can you please out a pastebin together of
>> hbck
>> >> > -details full output so that I can take a look at it.  Once I have
>> >> reviewed
>> >> > it, I should be able to provide you with a plan to out this back
>> >> together.
>> >> >
>> >> > On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <[EMAIL PROTECTED]>
>> >> wrote:
>> >> >
>> >> >> Hello Kevin,
>> >> >>
>> >> >> thanks for response. I clear it now and start cluster, but now i have
>> >> >> many other's regions :(
>> >> >> Here is output from hbase hbck -fixmeta
>> >> >> http://pastebin.com/HqsPVLMi
>> >> >>
>> >> >> Any next hint?
>> >> >>
>> >> >> Thanks a lot
>> >> >>
>> >> >> 2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
>> >> >> > Have you tried clearing out your Znode information?  Typically,
>> when I
>> >> >> have
>> >> >> > encountered a RIT, we will bring down HBase and go to the ZKcli and
>> >> clear
>> >> >> > out /hbase.  What do you see in the logs pertaining to the region?
>> >>  If it
>> >> >> > is a region that has bad hfiles or something like that you will
>> not be
>> >> >> able
>> >> >> > to easily get rid of it through clearing out the ZK.
>> >> >> >
>> >> >> > On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <
>> [EMAIL PROTECTED]>
>> >> >> wrote:
>> >> >> >
>> >> >> >> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
>> >> >> >>
>> >> >> >> I found this in master log http://pastebin.com/CYt5PZCL and this
>> >> line
>> >> >> >> are repeated for all region servers.
>> >> >> >>
>> >> >> >> Can someone help me please?
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Kevin O'Dell
>> >> >> > Customer Operations Engineer, Cloudera
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Save The World - http://www.worldcommunitygrid.org/
>> >> >>
>> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>> >> >>
>> >> >> LesTR
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Kevin O'Dell
>> >> > Customer Operations Engineer, Cloudera
>> >>
>> >>
>> >>
>> >> --
>> >> Save The World - http://www.worldcommunitygrid.org/
>> >> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR

Save The World - http://www.worldcommunitygrid.org/
http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR

LesTR

Lukas,

  I am not sure I understand what you are saying there.  What I thought you
did was:

hadoop fs -mv /hbase /tmp/hbase.old

Restart HBase

You should now have a clean HBase

hadoop fs -mv /tmp/hbase.old/<table_name>

Once you have moved all of the tables

./bin/hbase hbck -fixMeta -fixAssignments

This should put your hbase back into a happy state, if it doesn't not you
had underlying issues that need resolving.

On Mon, Oct 15, 2012 at 12:09 PM, Lukáš Drbal <[EMAIL PROTECTED]> wrote:

> Wow, its perfect "hack" but what about "exists" rows?
> For example when i have saved new version from row in /hbase i can't
> replace it from /hbase.old
>
> Thanks for time
>
> Lukas
>
> 2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
> > Lukas,
> >
> >   A little trick you can use is to just copy the table directories into
> > your new /hbase dir and then use:
> >
> > hbck -fixMeta -fixAssignments
> >
> > This will pull in the tables into your new empty META.  If once you do
> this
> > Regions get stuck in transition again, you have bigger problems.
> >
> > On Mon, Oct 15, 2012 at 11:48 AM, Lukáš Drbal <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hi again.
> >>
> >> > Does FSCK come back clean?  Are those the regions showing in
> transition?
> >> >  We are not going to be able to get a clear idea of what to do next
> until
> >> > we gather some more data.  At this point running repairs could put
> your
> >> > data in jeopardy.
> >>
> >>
> >> Ye, this is in transition, but from hadoop fsck are all ok and
> >> consistent - reported only 14 undereplicated blocks from another
> >> directory (not /hbase).
> >>
> >> I must bring cluster back online, so i move /hbase on hdfs to
> >> /hbase.old and start clear cluster with new tables :(
> >>
> >> Can you give me please any suggestion how i can put data from old
> >> files to new tables?
> >>
> >> Thanks
> >>
> >> My today's night will be very long :(
> >>
> >> Lukas
> >>
> >> >
> >> > On Mon, Oct 15, 2012 at 10:58 AM, Lukáš Drbal <[EMAIL PROTECTED]>
> >> wrote:
> >> >
> >> >> Kevin,
> >> >>
> >> >> Its more than one hour and region are still in transition.
> >> >> Here is output from hbck -detailed http://pastebin.com/f9wLx9LU
> >> >>
> >> >> Thanks for time
> >> >>
> >> >> Lukas
> >> >>
> >> >> 2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
> >> >> > Lukas,
> >> >> >
> >> >> >   Sure, how long has it been since the restart?  You will need to
> give
> >> >> the
> >> >> > regions time to transition and logs time to split.  Did the region
> >> >> > in transition properly?  Can you please out a pastebin together of
> >> hbck
> >> >> > -details full output so that I can take a look at it.  Once I have
> >> >> reviewed
> >> >> > it, I should be able to provide you with a plan to out this back
> >> >> together.
> >> >> >
> >> >> > On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <
> [EMAIL PROTECTED]>
> >> >> wrote:
> >> >> >
> >> >> >> Hello Kevin,
> >> >> >>
> >> >> >> thanks for response. I clear it now and start cluster, but now i
> have
> >> >> >> many other's regions :(
> >> >> >> Here is output from hbase hbck -fixmeta
> >> >> >> http://pastebin.com/HqsPVLMi
> >> >> >>
> >> >> >> Any next hint?
> >> >> >>
> >> >> >> Thanks a lot
> >> >> >>
> >> >> >> 2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
> >> >> >> > Have you tried clearing out your Znode information?  Typically,
> >> when I
> >> >> >> have
> >> >> >> > encountered a RIT, we will bring down HBase and go to the ZKcli
> and
> >> >> clear
> >> >> >> > out /hbase.  What do you see in the logs pertaining to the
> region?
> >> >>  If it
> >> >> >> > is a region that has bad hfiles or something like that you will
> >> not be
> >> >> >> able
> >> >> >> > to easily get rid of it through clearing out the ZK.
> >> >> >> >
> >> >> >> > On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <
> >> [EMAIL PROTECTED]>
> >> >> >> wrote:
> >> >> >> >
> >> >> >> >> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
> >> >> >> >>
> >> >> >> >> I found this in master log http://pastebin.com/CYt5PZCL and

Kevin O'Dell
Customer Operations Engineer, Cloudera

Sorry Kevin, english isn't my primary language.Iam so sorry.

What i need:
- i have a lot of data actualy in /hbase.old
- new data are saved to /hbase

and now i need "combine" all this data, but exists rows from /hbase
can't be rewrited by rows from /hbase.old.

Maybe i can do it like this:
hadoop fs -mv /hbase /hbase.old - this replace all exists old rows?

After this i can move /hbase.old to /hbase and run hbck.
Both with stopped cluster.

Lukas

2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
> Lukas,
>
>   I am not sure I understand what you are saying there.  What I thought you
> did was:
>
> hadoop fs -mv /hbase /tmp/hbase.old
>
> Restart HBase
>
> You should now have a clean HBase
>
> hadoop fs -mv /tmp/hbase.old/<table_name>
>
> Once you have moved all of the tables
>
> ./bin/hbase hbck -fixMeta -fixAssignments
>
> This should put your hbase back into a happy state, if it doesn't not you
> had underlying issues that need resolving.
>
> On Mon, Oct 15, 2012 at 12:09 PM, Lukáš Drbal <[EMAIL PROTECTED]> wrote:
>
>> Wow, its perfect "hack" but what about "exists" rows?
>> For example when i have saved new version from row in /hbase i can't
>> replace it from /hbase.old
>>
>> Thanks for time
>>
>> Lukas
>>
>> 2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
>> > Lukas,
>> >
>> >   A little trick you can use is to just copy the table directories into
>> > your new /hbase dir and then use:
>> >
>> > hbck -fixMeta -fixAssignments
>> >
>> > This will pull in the tables into your new empty META.  If once you do
>> this
>> > Regions get stuck in transition again, you have bigger problems.
>> >
>> > On Mon, Oct 15, 2012 at 11:48 AM, Lukáš Drbal <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >> Hi again.
>> >>
>> >> > Does FSCK come back clean?  Are those the regions showing in
>> transition?
>> >> >  We are not going to be able to get a clear idea of what to do next
>> until
>> >> > we gather some more data.  At this point running repairs could put
>> your
>> >> > data in jeopardy.
>> >>
>> >>
>> >> Ye, this is in transition, but from hadoop fsck are all ok and
>> >> consistent - reported only 14 undereplicated blocks from another
>> >> directory (not /hbase).
>> >>
>> >> I must bring cluster back online, so i move /hbase on hdfs to
>> >> /hbase.old and start clear cluster with new tables :(
>> >>
>> >> Can you give me please any suggestion how i can put data from old
>> >> files to new tables?
>> >>
>> >> Thanks
>> >>
>> >> My today's night will be very long :(
>> >>
>> >> Lukas
>> >>
>> >> >
>> >> > On Mon, Oct 15, 2012 at 10:58 AM, Lukáš Drbal <[EMAIL PROTECTED]>
>> >> wrote:
>> >> >
>> >> >> Kevin,
>> >> >>
>> >> >> Its more than one hour and region are still in transition.
>> >> >> Here is output from hbck -detailed http://pastebin.com/f9wLx9LU
>> >> >>
>> >> >> Thanks for time
>> >> >>
>> >> >> Lukas
>> >> >>
>> >> >> 2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
>> >> >> > Lukas,
>> >> >> >
>> >> >> >   Sure, how long has it been since the restart?  You will need to
>> give
>> >> >> the
>> >> >> > regions time to transition and logs time to split.  Did the region
>> >> >> > in transition properly?  Can you please out a pastebin together of
>> >> hbck
>> >> >> > -details full output so that I can take a look at it.  Once I have
>> >> >> reviewed
>> >> >> > it, I should be able to provide you with a plan to out this back
>> >> >> together.
>> >> >> >
>> >> >> > On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <
>> [EMAIL PROTECTED]>
>> >> >> wrote:
>> >> >> >
>> >> >> >> Hello Kevin,
>> >> >> >>
>> >> >> >> thanks for response. I clear it now and start cluster, but now i
>> have
>> >> >> >> many other's regions :(
>> >> >> >> Here is output from hbase hbck -fixmeta
>> >> >> >> http://pastebin.com/HqsPVLMi
>> >> >> >>
>> >> >> >> Any next hint?
>> >> >> >>
>> >> >> >> Thanks a lot
>> >> >> >>
>> >> >> >> 2012/10/15 Kevin O'dell <[EMAIL PROTECTED]>:
>> >> >> >> > Have you tried clearing out your Znode information?  Typically,

Save The World - http://www.worldcommunitygrid.org/
http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR

LesTR