Hi

In my flume agent with "Avro source-File Channel-HDFS sink"; I am
getting following exception:

*java.io.IOException: Failed to obtain lock for writing to the log. Try
increasing the log write timeout value or disabling it by setting it to
0. [channel=fileChannel]**
*
when I set my file roll interval to some high value (say 600 sec). It
works fine for smaller values like (180 sec).
I have also tried reducing batch size. I have tried setting both
"keep-alive" and "write-timeout" values of the channel
to 0 once and also to high value like 100 other time. But in both the
cases same exception appeared.

Regards,
Jagadish

This is my configuration:
============================================agent.sources = avro-collection-source
agent.channels = fileChannel
agent.sinks = hdfsSink fileSink

# For each one of the sources, the type is defined
agent.sources.avro-collection-source.type=avro
agent.sources.avro-collection-source.bind=10.0.17.3
agent.sources.avro-collection-source.port=10011
agent.sources.avro-collection-source.interceptors = ts
agent.sources.avro-collection-source.interceptors.ts.type = timestamp
# The channel can be defined as follows.
agent.sources.avro-collection-source.channels = fileChannel

# Each sink's type must be defined
agent.sinks.hdfsSink.type = hdfs
#agent.sinks.hdfsSink.hdfs.path=
hdfs://mltest2001.pubmatic.com/flume/experiments
agent.sinks.hdfsSink.hdfs.path=hdfs://mltest2001.pubmatic.com/flume/tracker3

#agent.sinks.hdfsSink.hdfs.fileType =DataStream
agent.sinks.hdfsSink.hdfs.fileType =CompressedStream
agent.sinks.hdfsSink.hdfs.filePrefix=adtrack_backup_sizeroll_%Y%m%d_%H%M%S_
#agent.sinks.hdfsSink.hdfs.filePrefix=adtrack_backup_
agent.sinks.hdfsSink.hdfs.rollSize=0
agent.sinks.hdfsSink.hdfs.codeC=bzip2
agent.sinks.hdfsSink.hdfs.rollCount=0
agent.sinks.hdfsSink.hdfs.batchSize=1000
#agent.sinks.hdfsSink.hdfs.writeFormat=Text
agent.sinks.hdfsSink.hdfs.rollInterval=600
#Define file sink
agent.sinks.fileSink.type = file_roll
agent.sinks.fileSink.sink.directory = /root/flume_sink

#Specify the channel the sink should use
agent.sinks.hdfsSink.channel= fileChannel
agent.channels.fileChannel.type=file
agent.channels.fileChannel.dataDirs=/root/flume_channel/dataDir13
agent.channels.fileChannel.checkpointDir=/root/flume_channel/checkpointDir13
agent.channels.fileChannel.write-timeout=30

Hi,

Could you  the full log on pastebin with us?

1) That error message is bad, setting it to 0 waits 0 time, not disabling it.

https://issues.apache.org/jira/browse/FLUME-1599

2) Do you know how full the channel is during the error? Or how many
events per second are going through?

Brock

On Fri, Sep 21, 2012 at 12:25 AM, Jagadish Bihani
<[EMAIL PROTECTED]> wrote:
> Hi
>
> In my flume agent with "Avro source-File Channel-HDFS sink"; I am getting
> following exception:
>
> java.io.IOException: Failed to obtain lock for writing to the log. Try
> increasing the log write timeout value or disabling it by setting it to 0.
> [channel=fileChannel]
>
> when I set my file roll interval to some high value (say 600 sec). It works
> fine for smaller values like (180 sec).
> I have also tried reducing batch size. I have tried setting both
> "keep-alive" and "write-timeout" values of the channel
> to 0 once and also to high value like 100 other time. But in both the cases
> same exception appeared.
>
> Regards,
> Jagadish
>
> This is my configuration:
> ============================================> agent.sources = avro-collection-source
> agent.channels = fileChannel
> agent.sinks = hdfsSink fileSink
>
> # For each one of the sources, the type is defined
> agent.sources.avro-collection-source.type=avro
> agent.sources.avro-collection-source.bind=10.0.17.3
> agent.sources.avro-collection-source.port=10011
> agent.sources.avro-collection-source.interceptors = ts
> agent.sources.avro-collection-source.interceptors.ts.type = timestamp
> # The channel can be defined as follows.
> agent.sources.avro-collection-source.channels = fileChannel
>
> # Each sink's type must be defined
> agent.sinks.hdfsSink.type = hdfs
> #agent.sinks.hdfsSink.hdfs.path> hdfs://mltest2001.pubmatic.com/flume/experiments
> agent.sinks.hdfsSink.hdfs.path=hdfs://mltest2001.pubmatic.com/flume/tracker3
>
> #agent.sinks.hdfsSink.hdfs.fileType =DataStream
> agent.sinks.hdfsSink.hdfs.fileType =CompressedStream
> agent.sinks.hdfsSink.hdfs.filePrefix=adtrack_backup_sizeroll_%Y%m%d_%H%M%S_
> #agent.sinks.hdfsSink.hdfs.filePrefix=adtrack_backup_
> agent.sinks.hdfsSink.hdfs.rollSize=0
> agent.sinks.hdfsSink.hdfs.codeC=bzip2
> agent.sinks.hdfsSink.hdfs.rollCount=0
> agent.sinks.hdfsSink.hdfs.batchSize=1000
> #agent.sinks.hdfsSink.hdfs.writeFormat=Text
> agent.sinks.hdfsSink.hdfs.rollInterval=600
>
>
> #Define file sink
> agent.sinks.fileSink.type = file_roll
> agent.sinks.fileSink.sink.directory = /root/flume_sink
>
> #Specify the channel the sink should use
> agent.sinks.hdfsSink.channel= fileChannel
> agent.channels.fileChannel.type=file
> agent.channels.fileChannel.dataDirs=/root/flume_channel/dataDir13
> agent.channels.fileChannel.checkpointDir=/root/flume_channel/checkpointDir13
> agent.channels.fileChannel.write-timeout=30
>

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/