Hi all -- we're currently on cdh3u3 (0.90.4 + patches).  I have one
table in our cluster which seems to functioning fine (gets/puts/scans
are all working), but for which no regions are listed on the UI.  The
table/regions exist in .META.  Other tables in the same cluster show
their regions list fine.  Seems like this might be a problem with
.META. or ZK, but would appreciate any pointers.

1) hbase hbck reports 2 "multiply assigned to region servers"
inconsistencies, but on a table different than the one I'm having
problems with.
2) The hbase master log shows this fragment when navigating to
table.jsp for the affected table:

2012-09-10 11:29:07,682 DEBUG org.apache.zookeeper.ClientCnxn: Reading
reply sessionid:0x1370e3604c49580, packet:: clientPath:null
serverPath:null finished:false header:: 10,4  replyHeader::
10,167713215,-101  request:: '/hbase/table/sessions,F  response::
2012-09-10 11:29:07,682 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKUtil:
hconnection-0x1370e3604c49580 Unable to get data of znode
/hbase/table/sessions because node does not exist (not an error)
2012-09-10 11:29:07,682 DEBUG
org.apache.hadoop.hbase.client.MetaScanner: Scanning .META. starting
at row=sessions,,00000000000000 for max=2147483647 rows

But since I see this "Unable to get data of znode" for all tables, my
assumption is that it's a red herring.  Any thoughts as how to debug
further, or why only this table would not show a region list?

Norbert

On Mon, Sep 10, 2012 at 8:33 AM, Norbert Burger
<[EMAIL PROTECTED]> wrote:
> Hi all -- we're currently on cdh3u3 (0.90.4 + patches).  I have one
> table in our cluster which seems to functioning fine (gets/puts/scans
> are all working), but for which no regions are listed on the UI.  The
> table/regions exist in .META.  Other tables in the same cluster show
> their regions list fine.  Seems like this might be a problem with
> .META. or ZK, but would appreciate any pointers.
>
> 1) hbase hbck reports 2 "multiply assigned to region servers"
> inconsistencies, but on a table different than the one I'm having
> problems with.
> 2) The hbase master log shows this fragment when navigating to
> table.jsp for the affected table:
>
> 2012-09-10 11:29:07,682 DEBUG org.apache.zookeeper.ClientCnxn: Reading
> reply sessionid:0x1370e3604c49580, packet:: clientPath:null
> serverPath:null finished:false header:: 10,4  replyHeader::
> 10,167713215,-101  request:: '/hbase/table/sessions,F  response::
> 2012-09-10 11:29:07,682 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKUtil:
> hconnection-0x1370e3604c49580 Unable to get data of znode
> /hbase/table/sessions because node does not exist (not an error)
> 2012-09-10 11:29:07,682 DEBUG
> org.apache.hadoop.hbase.client.MetaScanner: Scanning .META. starting
> at row=sessions,,00000000000000 for max=2147483647 rows
>
> But since I see this "Unable to get data of znode" for all tables, my
> assumption is that it's a red herring.  Any thoughts as how to debug
> further, or why only this table would not show a region list?
>

What happens if you scan .META. in shell?

hbase> scan ".META."

Does it all show?

(You might want to echo into a file so you can poke around after scan is done).

St.Ack

It scans .META. table just like any other table. I just tested it and it
produced the expected output.

Thanks,
Srinivas M
 On Sep 10, 2012 12:19 PM, "Stack" <[EMAIL PROTECTED]> wrote:

> On Mon, Sep 10, 2012 at 8:33 AM, Norbert Burger
> <[EMAIL PROTECTED]> wrote:
> > Hi all -- we're currently on cdh3u3 (0.90.4 + patches).  I have one
> > table in our cluster which seems to functioning fine (gets/puts/scans
> > are all working), but for which no regions are listed on the UI.  The
> > table/regions exist in .META.  Other tables in the same cluster show
> > their regions list fine.  Seems like this might be a problem with
> > .META. or ZK, but would appreciate any pointers.
> >
> > 1) hbase hbck reports 2 "multiply assigned to region servers"
> > inconsistencies, but on a table different than the one I'm having
> > problems with.
> > 2) The hbase master log shows this fragment when navigating to
> > table.jsp for the affected table:
> >
> > 2012-09-10 11:29:07,682 DEBUG org.apache.zookeeper.ClientCnxn: Reading
> > reply sessionid:0x1370e3604c49580, packet:: clientPath:null
> > serverPath:null finished:false header:: 10,4  replyHeader::
> > 10,167713215,-101  request:: '/hbase/table/sessions,F  response::
> > 2012-09-10 11:29:07,682 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKUtil:
> > hconnection-0x1370e3604c49580 Unable to get data of znode
> > /hbase/table/sessions because node does not exist (not an error)
> > 2012-09-10 11:29:07,682 DEBUG
> > org.apache.hadoop.hbase.client.MetaScanner: Scanning .META. starting
> > at row=sessions,,00000000000000 for max=2147483647 rows
> >
> > But since I see this "Unable to get data of znode" for all tables, my
> > assumption is that it's a red herring.  Any thoughts as how to debug
> > further, or why only this table would not show a region list?
> >
>
> What happens if you scan .META. in shell?
>
> hbase> scan ".META."
>
> Does it all show?
>
> (You might want to echo into a file so you can poke around after scan is
> done).
>
> St.Ack
>

On Mon, Sep 10, 2012 at 10:24 AM, Srinivas Mupparapu
<[EMAIL PROTECTED]> wrote:
> It scans .META. table just like any other table. I just tested it and it
> produced the expected output.
>

When you refresh the master UI, it makes a few lines in the master
log.  Are these the lines you posted?  Mind checking again?  What does
the Master UI page look like?  Complete?  Or is it cut off where its
should be listing regions (maybe look at html src?).

If shell can scan .META., odd that UI can't.  Lets try and figure the
difference.

St.Ack
> Thanks,
> Srinivas M
>  On Sep 10, 2012 12:19 PM, "Stack" <[EMAIL PROTECTED]> wrote:
>
>> On Mon, Sep 10, 2012 at 8:33 AM, Norbert Burger
>> <[EMAIL PROTECTED]> wrote:
>> > Hi all -- we're currently on cdh3u3 (0.90.4 + patches).  I have one
>> > table in our cluster which seems to functioning fine (gets/puts/scans
>> > are all working), but for which no regions are listed on the UI.  The
>> > table/regions exist in .META.  Other tables in the same cluster show
>> > their regions list fine.  Seems like this might be a problem with
>> > .META. or ZK, but would appreciate any pointers.
>> >
>> > 1) hbase hbck reports 2 "multiply assigned to region servers"
>> > inconsistencies, but on a table different than the one I'm having
>> > problems with.
>> > 2) The hbase master log shows this fragment when navigating to
>> > table.jsp for the affected table:
>> >
>> > 2012-09-10 11:29:07,682 DEBUG org.apache.zookeeper.ClientCnxn: Reading
>> > reply sessionid:0x1370e3604c49580, packet:: clientPath:null
>> > serverPath:null finished:false header:: 10,4  replyHeader::
>> > 10,167713215,-101  request:: '/hbase/table/sessions,F  response::
>> > 2012-09-10 11:29:07,682 DEBUG
>> > org.apache.hadoop.hbase.zookeeper.ZKUtil:
>> > hconnection-0x1370e3604c49580 Unable to get data of znode
>> > /hbase/table/sessions because node does not exist (not an error)
>> > 2012-09-10 11:29:07,682 DEBUG
>> > org.apache.hadoop.hbase.client.MetaScanner: Scanning .META. starting
>> > at row=sessions,,00000000000000 for max=2147483647 rows
>> >
>> > But since I see this "Unable to get data of znode" for all tables, my
>> > assumption is that it's a red herring.  Any thoughts as how to debug
>> > further, or why only this table would not show a region list?
>> >
>>
>> What happens if you scan .META. in shell?
>>
>> hbase> scan ".META."
>>
>> Does it all show?
>>
>> (You might want to echo into a file so you can poke around after scan is
>> done).
>>
>> St.Ack
>>

On Mon, Sep 10, 2012 at 1:24 PM, Srinivas Mupparapu
<[EMAIL PROTECTED]> wrote:
> It scans .META. table just like any other table. I just tested it and it
> produced the expected output.

I'm pretty sure Srinivas scanned .META. in his own environment, not mine.  ;-)

>  On Sep 10, 2012 12:19 PM, "Stack" <[EMAIL PROTECTED]> wrote:
>> What happens if you scan .META. in shell?
>>
>> hbase> scan ".META."
>>
>> Does it all show?

Thanks, Stack.  Strangely, all regions do show up in .META.  The table
in question has 256 regions, and all are listed as rowkeys in .META.
Perhaps there's a column missing from region definitions which is
preventing the UI from rendering the regions?  I'll dig through the
code, but are there specific columns known to be expected the UI?

Fwiw, this particular table was the result of a rename_table.rb
attempt that didn't go as smoothly as I would've liked.  I had to dig
through .META. and resolve inconsistencies.

Norbert

On Mon, Sep 10, 2012 at 10:33 AM, Norbert Burger
<[EMAIL PROTECTED]> wrote:
> On Mon, Sep 10, 2012 at 1:24 PM, Srinivas Mupparapu
> <[EMAIL PROTECTED]> wrote:
>> It scans .META. table just like any other table. I just tested it and it
>> produced the expected output.
>
> I'm pretty sure Srinivas scanned .META. in his own environment, not mine.  ;-)
>
>>  On Sep 10, 2012 12:19 PM, "Stack" <[EMAIL PROTECTED]> wrote:
>>> What happens if you scan .META. in shell?
>>>
>>> hbase> scan ".META."
>>>
>>> Does it all show?
>
> Thanks, Stack.  Strangely, all regions do show up in .META.  The table
> in question has 256 regions, and all are listed as rowkeys in .META.
> Perhaps there's a column missing from region definitions which is
> preventing the UI from rendering the regions?  I'll dig through the
> code, but are there specific columns known to be expected the UI?
>
> Fwiw, this particular table was the result of a rename_table.rb
> attempt that didn't go as smoothly as I would've liked.  I had to dig
> through .META. and resolve inconsistencies.
>

What version of hbase?  Can you disable and reenable the table?  When
you scan the table in shell, do you see 'status=offline'.
St.Ack

On Mon, Sep 10, 2012 at 1:37 PM, Stack <[EMAIL PROTECTED]> wrote:
> What version of hbase?

We're on cdh3u3, 0.90.4 + patches.

> Can you disable and reenable the table?

I will try disabling/re-enabling at the next opportunity.  Perhaps
that'll resolve that the issue, but this is a PROD cluster, so
unfortunately can't try right away.

> When you scan the table in shell, do you see 'status=offline'.

a) the master UI (table.jsp) shows the table, and reports enabled=true
b) is_enabled from the shell also reports true

Norbert

On Mon, Sep 10, 2012 at 10:50 AM, Norbert Burger
<[EMAIL PROTECTED]> wrote:
> On Mon, Sep 10, 2012 at 1:37 PM, Stack <[EMAIL PROTECTED]> wrote:
>> What version of hbase?
>
> We're on cdh3u3, 0.90.4 + patches.
>
>> Can you disable and reenable the table?
>
> I will try disabling/re-enabling at the next opportunity.  Perhaps
> that'll resolve that the issue, but this is a PROD cluster, so
> unfortunately can't try right away.
>
>> When you scan the table in shell, do you see 'status=offline'.

Thanks.  I was asking about the info:regioninfo column that prints out
the HRegionInfo for each region.  I was wondering if it included a
status=offline attribute.

You could try one region only and see if that makes a difference.

My guess is that this a vestige of the rename script.  You disabled
before using it (as it asks for at head of script).

St.Ack

On Mon, Sep 10, 2012 at 2:17 PM, Stack <[EMAIL PROTECTED]> wrote:
> Thanks.  I was asking about the info:regioninfo column that prints out
> the HRegionInfo for each region.  I was wondering if it included a
> status=offline attribute.
>
> You could try one region only and see if that makes a difference.

Hmmm... no status=offline anywhere in my dump of .META.  Will dig into
the code and try disable/enable when I get I chance.

Attached below is a copy of .META. for one of the problematic regions.
 As far as I can tell, it has all the required cols, and I don't see a
difference between this and a region which gets "displayed" correctly:

 sessions,,1342211893146.8 column=info:regioninfo,
timestamp=1342211893177, value=REGION => {NAME =>
 81ca12b4a6c9a7670bb7ef69b
'sessions,,1342211893146.881ca12b4a6c9a7670bb7ef69b3e5db4.', STARTKEY
=>
 3e5db4.                    '', ENDKEY => '01', ENCODED =>
881ca12b4a6c9a7670bb7ef69b3e5db4, TABLE                            > {{NAME => 'sessions', FAMILIES => [{NAME
=> 'event', BLOOMFILTER => 'RO
                           W', REPLICATION_SCOPE => '0', VERSIONS =>
'1', COMPRESSION => 'LZO', TTL
                           => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE                            > 'true'}]}}
 sessions,,1342211893146.8 column=info:server,
timestamp=1346979452245, value=aspen8hdp2.turner.com:
 81ca12b4a6c9a7670bb7ef69b 60020
 3e5db4.
 sessions,,1342211893146.8 column=info:serverstartcode,
timestamp=1346979452245, value=1346977435555
 81ca12b4a6c9a7670bb7ef69b
 3e5db4.

Thanks again,
Norbert

On Mon, Sep 10, 2012 at 12:05 PM, Norbert Burger
<[EMAIL PROTECTED]> wrote:
>
Mind putting up full listing in pastebin?

Let me have a look.

We could try a master restart too... so it refreshes its in-memory
state.  That might do it.

St.Ack

On Mon, Sep 10, 2012 at 3:29 PM, Stack <[EMAIL PROTECTED]> wrote:
> On Mon, Sep 10, 2012 at 12:05 PM, Norbert Burger
> <[EMAIL PROTECTED]> wrote:
>>
> Mind putting up full listing in pastebin?

Here's a link: http://pastebin.com/raw.php?i=4YhS8CpE.  The table in
question is called 'sessions', I did delete other tables' info from
this dump, as the .META. was quite large otherwise.

> We could try a master restart too... so it refreshes its in-memory
> state.  That might do it.

We've actually done this already, it hasn't seemed to resolve the situation.

Thanks,
Norbert