Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> : write-timeout   value tuning


+
Madhu Gmail 2013-04-08, 18:38
+
Brock Noland 2013-04-08, 18:42
+
Madhu Gmail 2013-04-08, 19:12
+
Brock Noland 2013-04-08, 19:15
Copy link to this message
-
Re: : write-timeout value tuning
Thanks Brock.

Madhu  Munagala
(214)679-2872

On Apr 8, 2013, at 2:15 PM, Brock Noland <[EMAIL PROTECTED]> wrote:

> The channel is checkpointing. Reducing the channels capacity or making whatever disk checkpointDir is on faster (dedicated disk, SSD, etc) would speed checkpointing.
>
>
> On Mon, Apr 8, 2013 at 2:12 PM, Madhu Gmail <[EMAIL PROTECTED]> wrote:
>> Thanks  Brock.
>>
>> But curious to know which other property setting causes this write- timeout issue.
>>
>> I have sink batch size as 20 and no batch size on the source side of the collector.   This agent acts as a collector for other flume agents which sends the events to the collector.
>>
>> Madhu  Munagala
>> (214)679-2872
>>
>> On Apr 8, 2013, at 1:42 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>>
>>> There is no harm in setting write-timeout to something like 30 seconds. In fact it probably makes sense to increase the default to 30 seconds.
>>>
>>>
>>> On Mon, Apr 8, 2013 at 1:38 PM, Madhu Gmail <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>>  
>>>>>
>>>>> I am getting below ERROR in flume agent(Acting as a collector)  which is receiving  log events from another  flume agent.
>>>>>
>>>>>  
>>>>>
>>>>> I have also copied my flume-conf.properties  at the end of this mail.
>>>>>
>>>>> Any idea how to tune  write-timeout  value.  
>>>>>
>>>>>  
>>>>>
>>>>>  
>>>>>
>>>>> 2013-04-05 13:17:33,197 ERROR org.apache.flume.SinkRunner: Unable to deliver event. Exception follows.
>>>>>
>>>>> org.apache.flume.ChannelException: Failed to obtain lock for writing to the log. Try increasing the log write timeout value. [channel=fc]
>>>>>
>>>>>                 at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:434)
>>>>>
>>>>>                 at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>
>>>>>                 at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:91)
>>>>>
>>>>>                 at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:189)
>>>>>
>>>>>                 at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>>>>
>>>>>                 at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>
>>>>>                 at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> 2013-04-05 13:17:33,427 INFO org.apache.flume.channel.file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1365169979081, queueSize: 0, queueHead: 362421
>>>>>
>>>>> 2013-04-05 13:17:34,233 INFO org.apache.flume.channel.file.LogFileV3: Updating log-14.meta currentPosition = 3818784, logWriteOrderID = 1365169979081
>>>>>
>>>>> 2013-04-05 13:17:34,294 INFO org.apache.flume.channel.file.Log: Updated checkpoint for file: /opt/sponge/flume/file-channel/dataDirs/log-14 position: 3818784 logWriteOrderID: 1365169979081
>>>>>
>>>>> 2013-04-05 13:17:34,294 DEBUG org.apache.flume.channel.file.Log: Rolling back 1365169950299
>>>>>
>>>>> 2013-04-05 13:17:34,296 ERROR org.apache.flume.source.AvroSource: Avro source S1: Unable to process event batch. Exception follows.
>>>>>
>>>>> org.apache.flume.ChannelException: Unable to put batch on required channel: FileChannel fc { dataDirs: [/opt/sponge/flume/file-channel/dataDirs] }
>>>>>
>>>>>                 at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
>>>>>
>>>>>                 at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:237)
>>>>>
>>>>>                 at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>>>
>>>>>                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>
>>>>>                 at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>
>>>>>                 at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>>>
>>>>>                 at org.apache.avro.ipc.Responder.respond(Responder.java:149)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB