Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> : write-timeout   value tuning


+
Madhu Gmail 2013-04-08, 18:38
+
Brock Noland 2013-04-08, 18:42
+
Madhu Gmail 2013-04-08, 19:12
Copy link to this message
-
Re: : write-timeout value tuning
The channel is checkpointing. Reducing the channels capacity or making
whatever disk checkpointDir is on faster (dedicated disk, SSD, etc) would
speed checkpointing.
On Mon, Apr 8, 2013 at 2:12 PM, Madhu Gmail <[EMAIL PROTECTED]>wrote:

> Thanks  Brock.
>
> But curious to know which other property setting causes this write-
> timeout issue.
>
> I have sink batch size as 20 and no batch size on the source side of the
> collector.   This agent acts as a collector for other flume agents which
> sends the events to the collector.
>
> Madhu  Munagala
> (214)679-2872
>
> On Apr 8, 2013, at 1:42 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>
> There is no harm in setting write-timeout to something like 30 seconds. In
> fact it probably makes sense to increase the default to 30 seconds.
>
>
> On Mon, Apr 8, 2013 at 1:38 PM, Madhu Gmail <[EMAIL PROTECTED]>wrote:
>
>>
>>  Hello,****
>>
>> ** **
>>
>> I am getting below ERROR in flume agent(Acting as a collector)  which is
>> receiving  log events from another  flume agent.****
>>
>> ** **
>>
>> I have also copied my flume-conf.properties  at the end of this mail.****
>>
>> Any idea how to tune  write-timeout  value.   ****
>>
>> ** **
>>
>> ** **
>>
>> 2013-04-05 13:17:33,197 ERROR org.apache.flume.SinkRunner: Unable to
>> deliver event. Exception follows.****
>>
>> org.apache.flume.ChannelException: Failed to obtain lock for writing to
>> the log. Try increasing the log write timeout value. [channel=fc]****
>>
>>                 at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:434)
>> ****
>>
>>                 at
>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>> ****
>>
>>                 at
>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:91)
>> ****
>>
>>                 at
>> org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:189)****
>>
>>                 at
>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>> ****
>>
>>                 at
>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)****
>>
>>                 at java.lang.Thread.run(Thread.java:662)****
>>
>> 2013-04-05 13:17:33,427 INFO
>> org.apache.flume.channel.file.EventQueueBackingStoreFile: Updating
>> checkpoint metadata: logWriteOrderID: 1365169979081, queueSize: 0,
>> queueHead: 362421****
>>
>> 2013-04-05 13:17:34,233 INFO org.apache.flume.channel.file.LogFileV3:
>> Updating log-14.meta currentPosition = 3818784, logWriteOrderID >> 1365169979081****
>>
>> 2013-04-05 13:17:34,294 INFO org.apache.flume.channel.file.Log: Updated
>> checkpoint for file: /opt/sponge/flume/file-channel/dataDirs/log-14
>> position: 3818784 logWriteOrderID: 1365169979081****
>>
>> 2013-04-05 13:17:34,294 DEBUG org.apache.flume.channel.file.Log: Rolling
>> back 1365169950299****
>>
>> 2013-04-05 13:17:34,296 ERROR org.apache.flume.source.AvroSource: Avro
>> source S1: Unable to process event batch. Exception follows.****
>>
>> org.apache.flume.ChannelException: Unable to put batch on required
>> channel: FileChannel fc { dataDirs:
>> [/opt/sponge/flume/file-channel/dataDirs] }****
>>
>>                 at
>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
>> ****
>>
>>                 at
>> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:237)****
>>
>>                 at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> Source)****
>>
>>                 at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> ****
>>
>>                 at java.lang.reflect.Method.invoke(Method.java:597)****
>>
>>                 at
>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>> ****
>>
>>                 at
>> org.apache.avro.ipc.Responder.respond(Responder.java:149)****
>>
>>                 at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
+
Madhu Gmail 2013-04-09, 01:25