Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - rpc call logging


Copy link to this message
-
Re: rpc call logging
Jack Levin 2011-04-11, 00:19
Ok, but I can't exactly use it for debugging unless I have some sort
of a viewer  - also, this is for writes, but mostly our issues are
reads.

-Jack

On Sun, Apr 10, 2011 at 4:58 PM, Stack <[EMAIL PROTECTED]> wrote:
> That's our wal log
>
>
>
> On Apr 10, 2011, at 13:46, Jack Levin <[EMAIL PROTECTED]> wrote:
>
>> I would love it to be exactly like mysqld.log where you save all your
>> queries there.
>>
>> -Jack
>>
>> On Sun, Apr 10, 2011 at 12:31 PM, Stack <[EMAIL PROTECTED]> wrote:
>>> They don't enough detail when you enable metrics on rpc?
>>>
>>> It should be easy enough hacking this in.  What would you like to see
>>> on the query Jack?  There'd be whether a Get/Delete/Put, etc., then
>>> what?   We'd have to give this logging a different name than that of
>>> the general regionserver class because you'd never want to run a
>>> production system with it on; you'd only want to enable it say via the
>>> UI for a short period of time, and then turn it off, because it'd be a
>>> drag on perf and it would make for reams of output (Consider server
>>> doing 1k hits a second, or in your case Jack, 100k hits a second --
>>> smile -- then we'd fill the disks in short order)
>>>
>>> St.Ack
>>>
>>>
>>> On Sat, Apr 9, 2011 at 7:09 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>>>> Well, counts would help a lot, but actual queries displayed even more
>>>> so... For example I just found a bug in our code that runs in infinite
>>>> loop across only specific pages that cause region server CPU to
>>>> overrun.   It would be easier for us to ID the bugs if we had all
>>>> queries logged.
>>>>
>>>> -Jack
>>>>
>>>> On Sat, Apr 9, 2011 at 6:02 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>>>> I think HBASE-3647 was a good start but Jack asked might be a little more.
>>>>> BTW 3647 isn't in 0.90.2
>>>>>
>>>>> Currently regionserver.jsp displays RegionServerMetrics for every region in
>>>>> an unsorted order w.r.t. read/write request counts. For any region server in
>>>>> production, there would be hundreds of regions, it is not realistic to spot
>>>>> hot regions in the current display.
>>>>> Do we need to add capability to sort the regions on regionserver.jsp
>>>>> according to read/write request counts ?
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Sat, Apr 9, 2011 at 1:59 PM, Stack <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> You mean per-region Jack?
>>>>>>
>>>>>> These went into TRUNK recently:
>>>>>>
>>>>>>   HBASE-3647  Distinguish read and write request count in region
>>>>>>               (Ted Yu via Stack)
>>>>>>   HBASE-3704  Show per region request count in table.jsp
>>>>>>               (Ted Yu via Stack)
>>>>>>
>>>>>> Would they help?
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>> On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>>>>>>> It would be super nice to have RPC query call logging per region
>>>>>>> server.  Sometimes we have 'hot' regions getting hit by users, and
>>>>>>> this drives the CPU on a particular RS server up, except we have no
>>>>>>> idea what table it might be.  I could rig the tcpdump on port 60020,
>>>>>>> and log things with AWK, but that would be super hacky.  An insight
>>>>>>> what traffic drives the cpu will allow us (or anyone else) model the
>>>>>>> tables to user patterns accordingly.
>>>>>>>
>>>>>>> thanks.
>>>>>>>
>>>>>>> -Jack
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>