Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - how to print the channel capacity


+
liuyongbo 2013-05-15, 05:37
+
Nitin Pawar 2013-05-15, 08:20
+
Nitin Pawar 2013-05-15, 08:46
Copy link to this message
-
Re: how to print the channel capacity
Nitin Pawar 2013-05-15, 08:49
here is one example for the capacity defining flow
https://cwiki.apache.org/FLUME/flume-ng-performance-measurements.html
On Wed, May 15, 2013 at 2:16 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:

> sorry pressed enter too soon
>
> as for your question: how many events a flume agent can hold?
> sorry but I don't think there is any direct answer to that.... .I may be
> very well wrong there as I am myself pretty new with flume
>
> there was a JIRA for the capacity of file channels FLUME-1571
>
>
> On Wed, May 15, 2013 at 1:50 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>
>> for maximum performance on your data flow two things which will matter
>> most are: the channel and the transaction batch size.
>> when you say losing data, are you using memory channel? or file channel?
>>
>> Flume can batch events. The batch size is the maximum number of events
>> that a sink or client will attempt to take from a channel in a single
>> transaction.
>>
>>  What is the channel type
>> do you have a slow sink so the # events written out are less than # event
>> incoming to channels so over time it piles up
>>
>> others may point out more things.
>> Also your flume conf and if you are seeing any errors on flume then that
>> will help people to find out the problem
>>
>>
>> On Wed, May 15, 2013 at 11:07 AM, liuyongbo <[EMAIL PROTECTED]> wrote:
>>
>>>  Hi:****
>>>
>>>          I’m using flume to pass log data to mongodb, but I find that
>>> some data lose when the pressure is in high level, so I want to know the
>>> max request that flume can hold and need to print the capacity.but I can
>>> not find the proper way to do this instead of change the source code. Any
>>> ideas?****
>>>
>>>          thanks****
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
> Nitin Pawar
>

--
Nitin Pawar
+
liuyongbo 2013-05-15, 11:39
+
Nitin Pawar 2013-05-15, 11:48
+
Paul Chavez 2013-05-15, 16:36
+
Matt Wise 2013-05-15, 17:10
+
Matt Wise 2013-05-15, 18:05
+
liuyongbo 2013-05-16, 08:06