It would be super nice to have RPC query call logging per region
server.  Sometimes we have 'hot' regions getting hit by users, and
this drives the CPU on a particular RS server up, except we have no
idea what table it might be.  I could rig the tcpdump on port 60020,
and log things with AWK, but that would be super hacky.  An insight
what traffic drives the cpu will allow us (or anyone else) model the
tables to user patterns accordingly.

thanks.

-Jack

You mean per-region Jack?

These went into TRUNK recently:

   HBASE-3647  Distinguish read and write request count in region
               (Ted Yu via Stack)
   HBASE-3704  Show per region request count in table.jsp
               (Ted Yu via Stack)

Would they help?

St.Ack

On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
> It would be super nice to have RPC query call logging per region
> server.  Sometimes we have 'hot' regions getting hit by users, and
> this drives the CPU on a particular RS server up, except we have no
> idea what table it might be.  I could rig the tcpdump on port 60020,
> and log things with AWK, but that would be super hacky.  An insight
> what traffic drives the cpu will allow us (or anyone else) model the
> tables to user patterns accordingly.
>
> thanks.
>
> -Jack
>

doesn't need in each case to be stored in a table but a rpc inside an
operating system could simply stream and broadcast the content

2011/4/9 Jack Levin <[EMAIL PROTECTED]>

> It would be super nice to have RPC query call logging per region
> server.  Sometimes we have 'hot' regions getting hit by users, and
> this drives the CPU on a particular RS server up, except we have no
> idea what table it might be.  I could rig the tcpdump on port 60020,
> and log things with AWK, but that would be super hacky.  An insight
> what traffic drives the cpu will allow us (or anyone else) model the
> tables to user patterns accordingly.
>
> thanks.
>
> -Jack
>

I think HBASE-3647 was a good start but Jack asked might be a little more.
BTW 3647 isn't in 0.90.2

Currently regionserver.jsp displays RegionServerMetrics for every region in
an unsorted order w.r.t. read/write request counts. For any region server in
production, there would be hundreds of regions, it is not realistic to spot
hot regions in the current display.
Do we need to add capability to sort the regions on regionserver.jsp
according to read/write request counts ?

Cheers

On Sat, Apr 9, 2011 at 1:59 PM, Stack <[EMAIL PROTECTED]> wrote:

> You mean per-region Jack?
>
> These went into TRUNK recently:
>
>   HBASE-3647  Distinguish read and write request count in region
>               (Ted Yu via Stack)
>   HBASE-3704  Show per region request count in table.jsp
>               (Ted Yu via Stack)
>
> Would they help?
>
> St.Ack
>
> On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
> > It would be super nice to have RPC query call logging per region
> > server.  Sometimes we have 'hot' regions getting hit by users, and
> > this drives the CPU on a particular RS server up, except we have no
> > idea what table it might be.  I could rig the tcpdump on port 60020,
> > and log things with AWK, but that would be super hacky.  An insight
> > what traffic drives the cpu will allow us (or anyone else) model the
> > tables to user patterns accordingly.
> >
> > thanks.
> >
> > -Jack
> >
>

Well, counts would help a lot, but actual queries displayed even more
so... For example I just found a bug in our code that runs in infinite
loop across only specific pages that cause region server CPU to
overrun.   It would be easier for us to ID the bugs if we had all
queries logged.

-Jack

On Sat, Apr 9, 2011 at 6:02 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> I think HBASE-3647 was a good start but Jack asked might be a little more.
> BTW 3647 isn't in 0.90.2
>
> Currently regionserver.jsp displays RegionServerMetrics for every region in
> an unsorted order w.r.t. read/write request counts. For any region server in
> production, there would be hundreds of regions, it is not realistic to spot
> hot regions in the current display.
> Do we need to add capability to sort the regions on regionserver.jsp
> according to read/write request counts ?
>
> Cheers
>
> On Sat, Apr 9, 2011 at 1:59 PM, Stack <[EMAIL PROTECTED]> wrote:
>
>> You mean per-region Jack?
>>
>> These went into TRUNK recently:
>>
>>   HBASE-3647  Distinguish read and write request count in region
>>               (Ted Yu via Stack)
>>   HBASE-3704  Show per region request count in table.jsp
>>               (Ted Yu via Stack)
>>
>> Would they help?
>>
>> St.Ack
>>
>> On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>> > It would be super nice to have RPC query call logging per region
>> > server.  Sometimes we have 'hot' regions getting hit by users, and
>> > this drives the CPU on a particular RS server up, except we have no
>> > idea what table it might be.  I could rig the tcpdump on port 60020,
>> > and log things with AWK, but that would be super hacky.  An insight
>> > what traffic drives the cpu will allow us (or anyone else) model the
>> > tables to user patterns accordingly.
>> >
>> > thanks.
>> >
>> > -Jack
>> >
>>
>

They don't enough detail when you enable metrics on rpc?

It should be easy enough hacking this in.  What would you like to see
on the query Jack?  There'd be whether a Get/Delete/Put, etc., then
what?   We'd have to give this logging a different name than that of
the general regionserver class because you'd never want to run a
production system with it on; you'd only want to enable it say via the
UI for a short period of time, and then turn it off, because it'd be a
drag on perf and it would make for reams of output (Consider server
doing 1k hits a second, or in your case Jack, 100k hits a second --
smile -- then we'd fill the disks in short order)

St.Ack
On Sat, Apr 9, 2011 at 7:09 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
> Well, counts would help a lot, but actual queries displayed even more
> so... For example I just found a bug in our code that runs in infinite
> loop across only specific pages that cause region server CPU to
> overrun.   It would be easier for us to ID the bugs if we had all
> queries logged.
>
> -Jack
>
> On Sat, Apr 9, 2011 at 6:02 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> I think HBASE-3647 was a good start but Jack asked might be a little more.
>> BTW 3647 isn't in 0.90.2
>>
>> Currently regionserver.jsp displays RegionServerMetrics for every region in
>> an unsorted order w.r.t. read/write request counts. For any region server in
>> production, there would be hundreds of regions, it is not realistic to spot
>> hot regions in the current display.
>> Do we need to add capability to sort the regions on regionserver.jsp
>> according to read/write request counts ?
>>
>> Cheers
>>
>> On Sat, Apr 9, 2011 at 1:59 PM, Stack <[EMAIL PROTECTED]> wrote:
>>
>>> You mean per-region Jack?
>>>
>>> These went into TRUNK recently:
>>>
>>>   HBASE-3647  Distinguish read and write request count in region
>>>               (Ted Yu via Stack)
>>>   HBASE-3704  Show per region request count in table.jsp
>>>               (Ted Yu via Stack)
>>>
>>> Would they help?
>>>
>>> St.Ack
>>>
>>> On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>>> > It would be super nice to have RPC query call logging per region
>>> > server.  Sometimes we have 'hot' regions getting hit by users, and
>>> > this drives the CPU on a particular RS server up, except we have no
>>> > idea what table it might be.  I could rig the tcpdump on port 60020,
>>> > and log things with AWK, but that would be super hacky.  An insight
>>> > what traffic drives the cpu will allow us (or anyone else) model the
>>> > tables to user patterns accordingly.
>>> >
>>> > thanks.
>>> >
>>> > -Jack
>>> >
>>>
>>
>

I would love it to be exactly like mysqld.log where you save all your
queries there.

-Jack

On Sun, Apr 10, 2011 at 12:31 PM, Stack <[EMAIL PROTECTED]> wrote:
> They don't enough detail when you enable metrics on rpc?
>
> It should be easy enough hacking this in.  What would you like to see
> on the query Jack?  There'd be whether a Get/Delete/Put, etc., then
> what?   We'd have to give this logging a different name than that of
> the general regionserver class because you'd never want to run a
> production system with it on; you'd only want to enable it say via the
> UI for a short period of time, and then turn it off, because it'd be a
> drag on perf and it would make for reams of output (Consider server
> doing 1k hits a second, or in your case Jack, 100k hits a second --
> smile -- then we'd fill the disks in short order)
>
> St.Ack
>
>
> On Sat, Apr 9, 2011 at 7:09 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>> Well, counts would help a lot, but actual queries displayed even more
>> so... For example I just found a bug in our code that runs in infinite
>> loop across only specific pages that cause region server CPU to
>> overrun.   It would be easier for us to ID the bugs if we had all
>> queries logged.
>>
>> -Jack
>>
>> On Sat, Apr 9, 2011 at 6:02 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>> I think HBASE-3647 was a good start but Jack asked might be a little more.
>>> BTW 3647 isn't in 0.90.2
>>>
>>> Currently regionserver.jsp displays RegionServerMetrics for every region in
>>> an unsorted order w.r.t. read/write request counts. For any region server in
>>> production, there would be hundreds of regions, it is not realistic to spot
>>> hot regions in the current display.
>>> Do we need to add capability to sort the regions on regionserver.jsp
>>> according to read/write request counts ?
>>>
>>> Cheers
>>>
>>> On Sat, Apr 9, 2011 at 1:59 PM, Stack <[EMAIL PROTECTED]> wrote:
>>>
>>>> You mean per-region Jack?
>>>>
>>>> These went into TRUNK recently:
>>>>
>>>>   HBASE-3647  Distinguish read and write request count in region
>>>>               (Ted Yu via Stack)
>>>>   HBASE-3704  Show per region request count in table.jsp
>>>>               (Ted Yu via Stack)
>>>>
>>>> Would they help?
>>>>
>>>> St.Ack
>>>>
>>>> On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>>>> > It would be super nice to have RPC query call logging per region
>>>> > server.  Sometimes we have 'hot' regions getting hit by users, and
>>>> > this drives the CPU on a particular RS server up, except we have no
>>>> > idea what table it might be.  I could rig the tcpdump on port 60020,
>>>> > and log things with AWK, but that would be super hacky.  An insight
>>>> > what traffic drives the cpu will allow us (or anyone else) model the
>>>> > tables to user patterns accordingly.
>>>> >
>>>> > thanks.
>>>> >
>>>> > -Jack
>>>> >
>>>>
>>>
>>
>

That's our wal log

On Apr 10, 2011, at 13:46, Jack Levin <[EMAIL PROTECTED]> wrote:

> I would love it to be exactly like mysqld.log where you save all your
> queries there.
>
> -Jack
>
> On Sun, Apr 10, 2011 at 12:31 PM, Stack <[EMAIL PROTECTED]> wrote:
>> They don't enough detail when you enable metrics on rpc?
>>
>> It should be easy enough hacking this in.  What would you like to see
>> on the query Jack?  There'd be whether a Get/Delete/Put, etc., then
>> what?   We'd have to give this logging a different name than that of
>> the general regionserver class because you'd never want to run a
>> production system with it on; you'd only want to enable it say via the
>> UI for a short period of time, and then turn it off, because it'd be a
>> drag on perf and it would make for reams of output (Consider server
>> doing 1k hits a second, or in your case Jack, 100k hits a second --
>> smile -- then we'd fill the disks in short order)
>>
>> St.Ack
>>
>>
>> On Sat, Apr 9, 2011 at 7:09 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>>> Well, counts would help a lot, but actual queries displayed even more
>>> so... For example I just found a bug in our code that runs in infinite
>>> loop across only specific pages that cause region server CPU to
>>> overrun.   It would be easier for us to ID the bugs if we had all
>>> queries logged.
>>>
>>> -Jack
>>>
>>> On Sat, Apr 9, 2011 at 6:02 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>>> I think HBASE-3647 was a good start but Jack asked might be a little more.
>>>> BTW 3647 isn't in 0.90.2
>>>>
>>>> Currently regionserver.jsp displays RegionServerMetrics for every region in
>>>> an unsorted order w.r.t. read/write request counts. For any region server in
>>>> production, there would be hundreds of regions, it is not realistic to spot
>>>> hot regions in the current display.
>>>> Do we need to add capability to sort the regions on regionserver.jsp
>>>> according to read/write request counts ?
>>>>
>>>> Cheers
>>>>
>>>> On Sat, Apr 9, 2011 at 1:59 PM, Stack <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> You mean per-region Jack?
>>>>>
>>>>> These went into TRUNK recently:
>>>>>
>>>>>   HBASE-3647  Distinguish read and write request count in region
>>>>>               (Ted Yu via Stack)
>>>>>   HBASE-3704  Show per region request count in table.jsp
>>>>>               (Ted Yu via Stack)
>>>>>
>>>>> Would they help?
>>>>>
>>>>> St.Ack
>>>>>
>>>>> On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>>>>>> It would be super nice to have RPC query call logging per region
>>>>>> server.  Sometimes we have 'hot' regions getting hit by users, and
>>>>>> this drives the CPU on a particular RS server up, except we have no
>>>>>> idea what table it might be.  I could rig the tcpdump on port 60020,
>>>>>> and log things with AWK, but that would be super hacky.  An insight
>>>>>> what traffic drives the cpu will allow us (or anyone else) model the
>>>>>> tables to user patterns accordingly.
>>>>>>
>>>>>> thanks.
>>>>>>
>>>>>> -Jack
>>>>>>
>>>>>
>>>>
>>>
>>

Ok, but I can't exactly use it for debugging unless I have some sort
of a viewer  - also, this is for writes, but mostly our issues are
reads.

-Jack

On Sun, Apr 10, 2011 at 4:58 PM, Stack <[EMAIL PROTECTED]> wrote:
> That's our wal log
>
>
>
> On Apr 10, 2011, at 13:46, Jack Levin <[EMAIL PROTECTED]> wrote:
>
>> I would love it to be exactly like mysqld.log where you save all your
>> queries there.
>>
>> -Jack
>>
>> On Sun, Apr 10, 2011 at 12:31 PM, Stack <[EMAIL PROTECTED]> wrote:
>>> They don't enough detail when you enable metrics on rpc?
>>>
>>> It should be easy enough hacking this in.  What would you like to see
>>> on the query Jack?  There'd be whether a Get/Delete/Put, etc., then
>>> what?   We'd have to give this logging a different name than that of
>>> the general regionserver class because you'd never want to run a
>>> production system with it on; you'd only want to enable it say via the
>>> UI for a short period of time, and then turn it off, because it'd be a
>>> drag on perf and it would make for reams of output (Consider server
>>> doing 1k hits a second, or in your case Jack, 100k hits a second --
>>> smile -- then we'd fill the disks in short order)
>>>
>>> St.Ack
>>>
>>>
>>> On Sat, Apr 9, 2011 at 7:09 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>>>> Well, counts would help a lot, but actual queries displayed even more
>>>> so... For example I just found a bug in our code that runs in infinite
>>>> loop across only specific pages that cause region server CPU to
>>>> overrun.   It would be easier for us to ID the bugs if we had all
>>>> queries logged.
>>>>
>>>> -Jack
>>>>
>>>> On Sat, Apr 9, 2011 at 6:02 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>>>> I think HBASE-3647 was a good start but Jack asked might be a little more.
>>>>> BTW 3647 isn't in 0.90.2
>>>>>
>>>>> Currently regionserver.jsp displays RegionServerMetrics for every region in
>>>>> an unsorted order w.r.t. read/write request counts. For any region server in
>>>>> production, there would be hundreds of regions, it is not realistic to spot
>>>>> hot regions in the current display.
>>>>> Do we need to add capability to sort the regions on regionserver.jsp
>>>>> according to read/write request counts ?
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Sat, Apr 9, 2011 at 1:59 PM, Stack <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> You mean per-region Jack?
>>>>>>
>>>>>> These went into TRUNK recently:
>>>>>>
>>>>>>   HBASE-3647  Distinguish read and write request count in region
>>>>>>               (Ted Yu via Stack)
>>>>>>   HBASE-3704  Show per region request count in table.jsp
>>>>>>               (Ted Yu via Stack)
>>>>>>
>>>>>> Would they help?
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>> On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>>>>>>> It would be super nice to have RPC query call logging per region
>>>>>>> server.  Sometimes we have 'hot' regions getting hit by users, and
>>>>>>> this drives the CPU on a particular RS server up, except we have no
>>>>>>> idea what table it might be.  I could rig the tcpdump on port 60020,
>>>>>>> and log things with AWK, but that would be super hacky.  An insight
>>>>>>> what traffic drives the cpu will allow us (or anyone else) model the
>>>>>>> tables to user patterns accordingly.
>>>>>>>
>>>>>>> thanks.
>>>>>>>
>>>>>>> -Jack
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>

This sounds like HBASE-2014: https://issues.apache.org/jira/browse/HBASE-2014

BTW apologies for the weird English in that issue, it appears I cut and pasted a request from our China development center without sufficient editing.

  - Andy