Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Lock contention in FileChannel


Copy link to this message
-
Re: Lock contention in FileChannel
Pankaj Gupta 2013-08-14, 19:12
With Null Sink, in the call stack I see a lot of these:
"SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
tid=0x00007feb18001000 nid=0x356b runnable [0x00007feb9e897000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.FileChannelImpl.force0(Native Method)
        at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:348)
        at
org.apache.flume.channel.file.LogFile$Writer.sync(LogFile.java:258)
        at
org.apache.flume.channel.file.LogFile$Writer.commit(LogFile.java:225)
        - locked <0x0000000518c97650> (a
org.apache.flume.channel.file.LogFileV3$Writer)
        at org.apache.flume.channel.file.Log.commit(Log.java:758)
        at org.apache.flume.channel.file.Log.commitTake(Log.java:640)
        at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:557)
        at
org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
        at org.apache.flume.sink.NullSink.process(NullSink.java:100)
        at
org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
        at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:662)
On Wed, Aug 14, 2013 at 11:57 AM, Pankaj Gupta <[EMAIL PROTECTED]>wrote:

> I tried increasing the dataDirs to 2 and 4 per disk but doesn't seem to
> help much. I then replaced the avro sink with null sinks and events are
> still filling up in the channel. I tried with both 2 and 4 dataDirs per
> disk and null sink, still don't get throughput higher than 1.5 MBps.
>
>
> On Tue, Aug 13, 2013 at 8:30 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>
>> Increasing the number of file channels will result in more checkpoints.
>> Therefore there will be more io than simply increasing the number of
>> dataDirs.  However, this might be a case where it'd be nice to relax the
>> file channel data consistency constraints a little to get increased
>> throughput. That feature does not exist at present.
>>
>>
>> On Tue, Aug 13, 2013 at 10:16 PM, Pankaj Gupta <[EMAIL PROTECTED]>wrote:
>>
>>> I did try increasing number of FileChannels. At 2 FileChannels per disk
>>> performance seemed to be 25% better. At 4 FileChannels per disk performance
>>> dropped to even below 1 FileChannel per disk. I will try increasing the
>>> dataDirs tomorrow.
>>>
>>>
>>> On Tue, Aug 13, 2013 at 8:06 PM, Brock Noland <[EMAIL PROTECTED]>wrote:
>>>
>>>> dataDirs is a comma separated list. Try 3-4 directories and then the
>>>> same test.
>>>> On Aug 13, 2013 9:58 PM, "Pankaj Gupta" <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Both disks were at around 15-25%.
>>>>>
>>>>>
>>>>> On Tue, Aug 13, 2013 at 7:54 PM, Brock Noland <[EMAIL PROTECTED]>wrote:
>>>>>
>>>>>> Gotcha. When you run tge test what is tye disk utilization
>>>>>> percentage? Iostat can be used for this.
>>>>>> On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <[EMAIL PROTECTED]>
>>>>>> wrote:
>>>>>>
>>>>>>> Those are the boxes we want to collect data from. They run flume and
>>>>>>> send data through their avro sinks to the avro source on this box. We are
>>>>>>> getting data at a pretty good rate and the problem is in fact that the
>>>>>>> events don't drain from the FileChannel fast enough and the channel fill
>>>>>>> percentage keeps getting higher.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <[EMAIL PROTECTED]>wrote:
>>>>>>>
>>>>>>>> What is sending the events to the avro source?
>>>>>>>> On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <[EMAIL PROTECTED]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Here's the config:
>>>>>>>>> # define channels, one for each disk
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.channels.ch1.type = FILE
>>>>>>>>> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
>>>>>>>>> agent1.channels.ch1.dataDirs = /flume1/data
>>>>>>>>> agent1.channels.ch1.maxFileSize = 375809638400
>>>>>>>>> agent1.channels.ch1.capacity = 75000000

*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | [EMAIL PROTECTED]

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
United States | Canada | United Kingdom | Germany
We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!