Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Lock contention in FileChannel


Copy link to this message
-
Re: Lock contention in FileChannel
You can increase the number of Avro Sinks active at a time, by increasing
the number of sink groups. This should give better throughput.
Hari
On Wed, Aug 14, 2013 at 12:34 PM, Pankaj Gupta <[EMAIL PROTECTED]>wrote:

> Null sink is actually holding up. I hadn't specified the batchSize so the
> default of 100 was being picked up. When I set batchSize to 4000 it started
> consuming all events and the channels are not filling up. So it seems that
> the problem is a combination of AvroSink with FileChannel. Memory Channel
> with Avro Sink works fine and FileChannel with null sink works fine.
>
>
> On Wed, Aug 14, 2013 at 12:12 PM, Pankaj Gupta <[EMAIL PROTECTED]>wrote:
>
>> With Null Sink, in the call stack I see a lot of these:
>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>> tid=0x00007feb18001000 nid=0x356b runnable [0x00007feb9e897000]
>>     java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.FileChannelImpl.force0(Native Method)
>>         at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:348)
>>         at
>> org.apache.flume.channel.file.LogFile$Writer.sync(LogFile.java:258)
>>         at
>> org.apache.flume.channel.file.LogFile$Writer.commit(LogFile.java:225)
>>         - locked <0x0000000518c97650> (a
>> org.apache.flume.channel.file.LogFileV3$Writer)
>>         at org.apache.flume.channel.file.Log.commit(Log.java:758)
>>         at org.apache.flume.channel.file.Log.commitTake(Log.java:640)
>>         at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:557)
>>         at
>> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>>         at org.apache.flume.sink.NullSink.process(NullSink.java:100)
>>         at
>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>         at
>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>         at java.lang.Thread.run(Thread.java:662)
>>
>>
>>
>>
>> On Wed, Aug 14, 2013 at 11:57 AM, Pankaj Gupta <[EMAIL PROTECTED]>wrote:
>>
>>> I tried increasing the dataDirs to 2 and 4 per disk but doesn't seem to
>>> help much. I then replaced the avro sink with null sinks and events are
>>> still filling up in the channel. I tried with both 2 and 4 dataDirs per
>>> disk and null sink, still don't get throughput higher than 1.5 MBps.
>>>
>>>
>>> On Tue, Aug 13, 2013 at 8:30 PM, Brock Noland <[EMAIL PROTECTED]>wrote:
>>>
>>>> Increasing the number of file channels will result in more checkpoints.
>>>> Therefore there will be more io than simply increasing the number of
>>>> dataDirs.  However, this might be a case where it'd be nice to relax the
>>>> file channel data consistency constraints a little to get increased
>>>> throughput. That feature does not exist at present.
>>>>
>>>>
>>>> On Tue, Aug 13, 2013 at 10:16 PM, Pankaj Gupta <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> I did try increasing number of FileChannels. At 2 FileChannels per
>>>>> disk performance seemed to be 25% better. At 4 FileChannels per disk
>>>>> performance dropped to even below 1 FileChannel per disk. I will try
>>>>> increasing the dataDirs tomorrow.
>>>>>
>>>>>
>>>>> On Tue, Aug 13, 2013 at 8:06 PM, Brock Noland <[EMAIL PROTECTED]>wrote:
>>>>>
>>>>>> dataDirs is a comma separated list. Try 3-4 directories and then the
>>>>>> same test.
>>>>>> On Aug 13, 2013 9:58 PM, "Pankaj Gupta" <[EMAIL PROTECTED]>
>>>>>> wrote:
>>>>>>
>>>>>>> Both disks were at around 15-25%.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 13, 2013 at 7:54 PM, Brock Noland <[EMAIL PROTECTED]>wrote:
>>>>>>>
>>>>>>>> Gotcha. When you run tge test what is tye disk utilization
>>>>>>>> percentage? Iostat can be used for this.
>>>>>>>> On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <[EMAIL PROTECTED]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Those are the boxes we want to collect data from. They run flume
>>>>>>>>> and send data through their avro sinks to the avro source on this box. We