Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> rpc call logging


Copy link to this message
-
Re: rpc call logging
Ok, but I can't exactly use it for debugging unless I have some sort
of a viewer  - also, this is for writes, but mostly our issues are
reads.

-Jack

On Sun, Apr 10, 2011 at 4:58 PM, Stack <[EMAIL PROTECTED]> wrote:
> That's our wal log
>
>
>
> On Apr 10, 2011, at 13:46, Jack Levin <[EMAIL PROTECTED]> wrote:
>
>> I would love it to be exactly like mysqld.log where you save all your
>> queries there.
>>
>> -Jack
>>
>> On Sun, Apr 10, 2011 at 12:31 PM, Stack <[EMAIL PROTECTED]> wrote:
>>> They don't enough detail when you enable metrics on rpc?
>>>
>>> It should be easy enough hacking this in.  What would you like to see
>>> on the query Jack?  There'd be whether a Get/Delete/Put, etc., then
>>> what?   We'd have to give this logging a different name than that of
>>> the general regionserver class because you'd never want to run a
>>> production system with it on; you'd only want to enable it say via the
>>> UI for a short period of time, and then turn it off, because it'd be a
>>> drag on perf and it would make for reams of output (Consider server
>>> doing 1k hits a second, or in your case Jack, 100k hits a second --
>>> smile -- then we'd fill the disks in short order)
>>>
>>> St.Ack
>>>
>>>
>>> On Sat, Apr 9, 2011 at 7:09 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>>>> Well, counts would help a lot, but actual queries displayed even more
>>>> so... For example I just found a bug in our code that runs in infinite
>>>> loop across only specific pages that cause region server CPU to
>>>> overrun.   It would be easier for us to ID the bugs if we had all
>>>> queries logged.
>>>>
>>>> -Jack
>>>>
>>>> On Sat, Apr 9, 2011 at 6:02 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>>>> I think HBASE-3647 was a good start but Jack asked might be a little more.
>>>>> BTW 3647 isn't in 0.90.2
>>>>>
>>>>> Currently regionserver.jsp displays RegionServerMetrics for every region in
>>>>> an unsorted order w.r.t. read/write request counts. For any region server in
>>>>> production, there would be hundreds of regions, it is not realistic to spot
>>>>> hot regions in the current display.
>>>>> Do we need to add capability to sort the regions on regionserver.jsp
>>>>> according to read/write request counts ?
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Sat, Apr 9, 2011 at 1:59 PM, Stack <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> You mean per-region Jack?
>>>>>>
>>>>>> These went into TRUNK recently:
>>>>>>
>>>>>>   HBASE-3647  Distinguish read and write request count in region
>>>>>>               (Ted Yu via Stack)
>>>>>>   HBASE-3704  Show per region request count in table.jsp
>>>>>>               (Ted Yu via Stack)
>>>>>>
>>>>>> Would they help?
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>> On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>>>>>>> It would be super nice to have RPC query call logging per region
>>>>>>> server.  Sometimes we have 'hot' regions getting hit by users, and
>>>>>>> this drives the CPU on a particular RS server up, except we have no
>>>>>>> idea what table it might be.  I could rig the tcpdump on port 60020,
>>>>>>> and log things with AWK, but that would be super hacky.  An insight
>>>>>>> what traffic drives the cpu will allow us (or anyone else) model the
>>>>>>> tables to user patterns accordingly.
>>>>>>>
>>>>>>> thanks.
>>>>>>>
>>>>>>> -Jack
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB