Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Lock contention in FileChannel


+
Pankaj Gupta 2013-08-13, 23:13
+
Hari Shreedharan 2013-08-13, 23:39
+
Pankaj Gupta 2013-08-14, 00:01
+
Hari Shreedharan 2013-08-14, 00:14
+
Brock Noland 2013-08-14, 00:51
+
Pankaj Gupta 2013-08-14, 02:06
+
Hari Shreedharan 2013-08-14, 02:18
+
Brock Noland 2013-08-14, 02:22
+
Pankaj Gupta 2013-08-14, 02:33
+
Brock Noland 2013-08-14, 02:41
+
Pankaj Gupta 2013-08-14, 02:46
+
Brock Noland 2013-08-14, 02:54
+
Pankaj Gupta 2013-08-14, 02:57
+
Brock Noland 2013-08-14, 03:06
+
Pankaj Gupta 2013-08-14, 03:16
+
Brock Noland 2013-08-14, 03:30
+
Pankaj Gupta 2013-08-14, 18:57
+
Pankaj Gupta 2013-08-14, 19:12
Copy link to this message
-
Re: Lock contention in FileChannel
Null sink is actually holding up. I hadn't specified the batchSize so the
default of 100 was being picked up. When I set batchSize to 4000 it started
consuming all events and the channels are not filling up. So it seems that
the problem is a combination of AvroSink with FileChannel. Memory Channel
with Avro Sink works fine and FileChannel with null sink works fine.
On Wed, Aug 14, 2013 at 12:12 PM, Pankaj Gupta <[EMAIL PROTECTED]>wrote:

> With Null Sink, in the call stack I see a lot of these:
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
> tid=0x00007feb18001000 nid=0x356b runnable [0x00007feb9e897000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.FileChannelImpl.force0(Native Method)
>         at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:348)
>         at
> org.apache.flume.channel.file.LogFile$Writer.sync(LogFile.java:258)
>         at
> org.apache.flume.channel.file.LogFile$Writer.commit(LogFile.java:225)
>         - locked <0x0000000518c97650> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
>         at org.apache.flume.channel.file.Log.commit(Log.java:758)
>         at org.apache.flume.channel.file.Log.commitTake(Log.java:640)
>         at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:557)
>         at
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>         at org.apache.flume.sink.NullSink.process(NullSink.java:100)
>         at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:662)
>
>
>
>
> On Wed, Aug 14, 2013 at 11:57 AM, Pankaj Gupta <[EMAIL PROTECTED]>wrote:
>
>> I tried increasing the dataDirs to 2 and 4 per disk but doesn't seem to
>> help much. I then replaced the avro sink with null sinks and events are
>> still filling up in the channel. I tried with both 2 and 4 dataDirs per
>> disk and null sink, still don't get throughput higher than 1.5 MBps.
>>
>>
>> On Tue, Aug 13, 2013 at 8:30 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>>
>>> Increasing the number of file channels will result in more checkpoints.
>>> Therefore there will be more io than simply increasing the number of
>>> dataDirs.  However, this might be a case where it'd be nice to relax the
>>> file channel data consistency constraints a little to get increased
>>> throughput. That feature does not exist at present.
>>>
>>>
>>> On Tue, Aug 13, 2013 at 10:16 PM, Pankaj Gupta <[EMAIL PROTECTED]>wrote:
>>>
>>>> I did try increasing number of FileChannels. At 2 FileChannels per disk
>>>> performance seemed to be 25% better. At 4 FileChannels per disk performance
>>>> dropped to even below 1 FileChannel per disk. I will try increasing the
>>>> dataDirs tomorrow.
>>>>
>>>>
>>>> On Tue, Aug 13, 2013 at 8:06 PM, Brock Noland <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> dataDirs is a comma separated list. Try 3-4 directories and then the
>>>>> same test.
>>>>> On Aug 13, 2013 9:58 PM, "Pankaj Gupta" <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> Both disks were at around 15-25%.
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 13, 2013 at 7:54 PM, Brock Noland <[EMAIL PROTECTED]>wrote:
>>>>>>
>>>>>>> Gotcha. When you run tge test what is tye disk utilization
>>>>>>> percentage? Iostat can be used for this.
>>>>>>> On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <[EMAIL PROTECTED]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Those are the boxes we want to collect data from. They run flume
>>>>>>>> and send data through their avro sinks to the avro source on this box. We
>>>>>>>> are getting data at a pretty good rate and the problem is in fact that the
>>>>>>>> events don't drain from the FileChannel fast enough and the channel fill
>>>>>>>> percentage keeps getting higher.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <[EMAIL PROTECTED]>wrote:

*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | [EMAIL PROTECTED]

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
United States | Canada | United Kingdom | Germany
We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!
+
Hari Shreedharan 2013-08-14, 19:43
+
Pankaj Gupta 2013-08-14, 19:59
+
Pankaj Gupta 2013-08-15, 06:04
+
Hari Shreedharan 2013-08-14, 19:04
+
Pankaj Gupta 2013-08-14, 02:16
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB