Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Need for UDP / Multicast Source


+
Andrew Otto 2013-01-14, 17:29
+
Hari Shreedharan 2013-01-14, 17:37
+
Alexander Alten-Lorenz 2013-01-14, 17:43
+
Andrew Otto 2013-01-14, 18:01
+
Andrew Otto 2013-01-15, 19:31
+
Andrew Otto 2013-01-16, 21:22
+
Brock Noland 2013-01-16, 21:36
+
Andrew Otto 2013-01-16, 22:30
+
Brock Noland 2013-01-16, 22:34
+
Hari Shreedharan 2013-01-16, 22:47
+
Andrew Otto 2013-01-16, 23:03
+
Hari Shreedharan 2013-01-16, 23:09
+
Bhaskar V. Karambelkar 2013-01-17, 01:21
+
Andrew Otto 2013-01-17, 15:34
+
Andrew Otto 2013-01-17, 16:26
+
Andrew Otto 2013-01-17, 17:36
+
Jeff Lord 2013-01-17, 17:59
Copy link to this message
-
Re: Need for UDP / Multicast Source
Brock Noland 2013-01-17, 18:04
Yeah what jeff said. It would be interesting to know which component
cannot keep up, the source or sink. If the sink cannot keep up you'll
see a growing channel size.

I have written something similar to read events via UDP before. I
found that because UDP can so easily drop data, I needed a thread
dedicated to reading the events and then immediately hand them off
another thread to do anything interesting. It's possible you are in
this scenario.

On Thu, Jan 17, 2013 at 9:59 AM, Jeff Lord <[EMAIL PROTECTED]> wrote:
> Hi Andrew,
>
> You may try lowering transactionCapacity here.
> The transactionCapacity should be set to the value of the largest batch size
> that will be used to store or remove events from that channel. You currently
> have it equal to the capacity of the channel. So essentially the channel
> *could be* filled with one transaction depending on how you are batching
> with your client.
>
> Also it may be useful to turn up jmx monitoring and watch the channel
> counters using jconsole. This way you can see exactly how many events are
> placed in the channel.
>
> To do this you will need to Set the following Java system properties located
> at,
> /etc/flume-ng/conf/flume-env.sh.
>
> com.sun.management.jmxremote
> com.sun.management.jmxremote.port=8081
> com.sun.management.jmxremote.authenticate=false
> com.sun.management.jmxremote.ssl=false
>
> You should than be able to connect with
> jconsole hostname:8081
>
> -Jeff
>
>
>
>
> On Thu, Jan 17, 2013 at 9:36 AM, Andrew Otto <[EMAIL PROTECTED]> wrote:
>>
>> > I took those two border sequence numbers and extracted all of the log
>> > lines generated by cp1044 on the nearby host (not using Flume).  I should be
>> > able to compare the number of lines here with the number of lines in the 12
>> > files I extracted from HDFS and Flume.  If they are the same, then Flume and
>> > UDPSource is working!
>>
>> Oh, I meant to link to a Gist with my current flume.conf and the commands
>> I executed to investigate this.  Here it is:
>>
>> https://gist.github.com/4557178
>>
>>
>>
>

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
+
Andrew Otto 2013-01-17, 18:56
+
Andrew Otto 2013-01-17, 17:33