Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Lock contention in FileChannel


Copy link to this message
-
Re: Lock contention in FileChannel
Pankaj Gupta 2013-08-14, 19:34
Null sink is actually holding up. I hadn't specified the batchSize so the
default of 100 was being picked up. When I set batchSize to 4000 it started
consuming all events and the channels are not filling up. So it seems that
the problem is a combination of AvroSink with FileChannel. Memory Channel
with Avro Sink works fine and FileChannel with null sink works fine.
On Wed, Aug 14, 2013 at 12:12 PM, Pankaj Gupta <[EMAIL PROTECTED]>wrote:

> With Null Sink, in the call stack I see a lot of these:
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
> tid=0x00007feb18001000 nid=0x356b runnable [0x00007feb9e897000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.FileChannelImpl.force0(Native Method)
>         at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:348)
>         at
> org.apache.flume.channel.file.LogFile$Writer.sync(LogFile.java:258)
>         at
> org.apache.flume.channel.file.LogFile$Writer.commit(LogFile.java:225)
>         - locked <0x0000000518c97650> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
>         at org.apache.flume.channel.file.Log.commit(Log.java:758)
>         at org.apache.flume.channel.file.Log.commitTake(Log.java:640)
>         at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:557)
>         at
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>         at org.apache.flume.sink.NullSink.process(NullSink.java:100)
>         at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:662)
>
>
>
>
> On Wed, Aug 14, 2013 at 11:57 AM, Pankaj Gupta <[EMAIL PROTECTED]>wrote:
>
>> I tried increasing the dataDirs to 2 and 4 per disk but doesn't seem to
>> help much. I then replaced the avro sink with null sinks and events are
>> still filling up in the channel. I tried with both 2 and 4 dataDirs per
>> disk and null sink, still don't get throughput higher than 1.5 MBps.
>>
>>
>> On Tue, Aug 13, 2013 at 8:30 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>>
>>> Increasing the number of file channels will result in more checkpoints.
>>> Therefore there will be more io than simply increasing the number of
>>> dataDirs.  However, this might be a case where it'd be nice to relax the
>>> file channel data consistency constraints a little to get increased
>>> throughput. That feature does not exist at present.
>>>
>>>
>>> On Tue, Aug 13, 2013 at 10:16 PM, Pankaj Gupta <[EMAIL PROTECTED]>wrote:
>>>
>>>> I did try increasing number of FileChannels. At 2 FileChannels per disk
>>>> performance seemed to be 25% better. At 4 FileChannels per disk performance
>>>> dropped to even below 1 FileChannel per disk. I will try increasing the
>>>> dataDirs tomorrow.
>>>>
>>>>
>>>> On Tue, Aug 13, 2013 at 8:06 PM, Brock Noland <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> dataDirs is a comma separated list. Try 3-4 directories and then the
>>>>> same test.
>>>>> On Aug 13, 2013 9:58 PM, "Pankaj Gupta" <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> Both disks were at around 15-25%.
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 13, 2013 at 7:54 PM, Brock Noland <[EMAIL PROTECTED]>wrote:
>>>>>>
>>>>>>> Gotcha. When you run tge test what is tye disk utilization
>>>>>>> percentage? Iostat can be used for this.
>>>>>>> On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <[EMAIL PROTECTED]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Those are the boxes we want to collect data from. They run flume
>>>>>>>> and send data through their avro sinks to the avro source on this box. We
>>>>>>>> are getting data at a pretty good rate and the problem is in fact that the
>>>>>>>> events don't drain from the FileChannel fast enough and the channel fill
>>>>>>>> percentage keeps getting higher.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <[EMAIL PROTECTED]>wrote:

*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | [EMAIL PROTECTED]

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
United States | Canada | United Kingdom | Germany
We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!