Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> flood of "outstanding proposal" messages


Copy link to this message
-
Re: flood of "outstanding proposal" messages
Thanks very much, good to know.

On 07/07/2011 02:34 PM, Patrick Hunt wrote:
> When you are stressing the system outstanding proposals can backup,
> the entirety of which will be logged at TRACE level each time an ack
> is processed. Alone this should not cause any problems other than the
> large logs you highlighted in your original post.
>
> Patrick
>
> On Thu, Jul 7, 2011 at 2:22 PM, Jeremy Stribling<[EMAIL PROTECTED]>  wrote:
>    
>> Yes, thanks, we definitely will.  During this test we were trying to track
>> down some other problem we were having, which is why the debugging level is
>> turned up.  Are you saying that whatever underlying issue caused these
>> hundreds of megabytes of messages would only happen in TRACE mode?
>>
>> On 07/07/2011 02:12 PM, Patrick Hunt wrote:
>>      
>>> Hi Jeremy, you're running with TRACE logging turned on which can be
>>> very verbose, I'd suggest you run with INFO level in normal
>>> circumstances.
>>>
>>> Patrick
>>>
>>> On Thu, Jul 7, 2011 at 10:40 AM, Jeremy Stribling<[EMAIL PROTECTED]>
>>>   wrote:
>>>
>>>        
>>>> Has anyone ever seen Zookeeper explode with a flood of "outstanding
>>>> proposal" messages?  I'm running 3.3.3* with three nodes (under fairly
>>>> stressful client load), and all of the sudden one of the nodes' logs
>>>> starts
>>>> filling up with these messages:
>>>>
>>>> 2011-07-06 18:37:51,161 628684 [ProcessThread:-1] TRACE
>>>> org.apache.zookeeper.server.PrepRequestProcessor  -
>>>> :Psessionid:0x5d310234eaf10002 type:delete cxid:0x4e15574f
>>>> zxid:0xfffffffffffffffe txntype:unknown
>>>>
>>>> reqpath:/zkrsm/cpt-0000000000000009-000000000000afc8/0000000000000009_record0000003293
>>>> 2011-07-06 18:37:51,162 628685 [ProcessThread:-1] DEBUG
>>>> org.apache.zookeeper.server.quorum.CommitProcessor  - Processing
>>>> request::
>>>> sessionid:0x5d310234eaf10002 type:delete cxid:0x4e15574f zxid:0x40000121c
>>>> txntype:2 reqpath:n/a
>>>> 2011-07-06 18:37:51,162 628685 [SyncThread:177] TRACE
>>>> org.apache.zookeeper.server.quorum.Leader  - Ack zxid: 0x4000011fb
>>>> 2011-07-06 18:37:51,162 628685 [SyncThread:177] TRACE
>>>> org.apache.zookeeper.server.quorum.Leader  - outstanding proposal:
>>>> 0x400001204
>>>> 2011-07-06 18:37:51,162 628685 [SyncThread:177] TRACE
>>>> org.apache.zookeeper.server.quorum.Leader  - outstanding proposal:
>>>> 0x40000120d
>>>> 2011-07-06 18:37:51,162 628685 [SyncThread:177] TRACE
>>>> org.apache.zookeeper.server.quorum.Leader  - outstanding proposal:
>>>> 0x400001207
>>>> 2011-07-06 18:37:51,162 628685 [SyncThread:177] TRACE
>>>> org.apache.zookeeper.server.quorum.Leader  - outstanding proposal:
>>>> 0x400001215
>>>> 2011-07-06 18:37:51,162 628685 [SyncThread:177] TRACE
>>>> org.apache.zookeeper.server.quorum.Leader  - outstanding proposal:
>>>> 0x4000011fe
>>>> 2011-07-06 18:37:51,162 628685 [SyncThread:177] TRACE
>>>> org.apache.zookeeper.server.quorum.Leader  - outstanding proposal:
>>>> 0x400001208
>>>> 2011-07-06 18:37:51,162 628685 [SyncThread:177] TRACE
>>>> org.apache.zookeeper.server.quorum.Leader  - outstanding proposal:
>>>> 0x400001214
>>>> 2011-07-06 18:37:51,162 628685 [SyncThread:177] TRACE
>>>> org.apache.zookeeper.server.quorum.Leader  - outstanding proposal:
>>>> 0x400001219
>>>> 2011-07-06 18:37:51,162 628685 [SyncThread:177] TRACE
>>>> org.apache.zookeeper.server.quorum.Leader  - outstanding proposal:
>>>> 0x400001206
>>>> 2011-07-06 18:37:51,162 628685 [SyncThread:177] TRACE
>>>> org.apache.zookeeper.server.quorum.Leader  - outstanding proposal:
>>>> 0x400001201
>>>> 2011-07-06 18:37:51,162 628685 [SyncThread:177] TRACE
>>>> org.apache.zookeeper.server.quorum.Leader  - outstanding proposal:
>>>> 0x400001213
>>>> 2011-07-06 18:37:51,162 628685 [SyncThread:177] TRACE
>>>> org.apache.zookeeper.server.quorum.Leader  - outstanding proposal:
>>>> 0x4000011ff
>>>> 2011-07-06 18:37:51,162 628685 [SyncThread:177] TRACE
>>>> org.apache.zookeeper.server.quorum.Leader  - outstanding proposal:
>>>> 0x40000121a
>>>> 2011-07-06 18:37:51,162 628685 [SyncThread:177] TRACE
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB