Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Dropped events


Copy link to this message
-
Re: Dropped events
Your channel capacity is set to 1. Are you sure you really want that? If a null event is returned by the channel, then you should commit and close the transaction, optionally backoff (usually, this is when a batch was empty), and then try again. You should probably also use a higher transaction capacity.  
Hari  

--  
Hari Shreedharan
On Tuesday, March 12, 2013 at 11:03 AM, Vikram Kulkarni wrote:

>  
> I have my custom Source and Sink that I have hooked with a memory channel but I am noticing that it is not very consistent. Even after sending many events to the Source the Sink’s event is still null. It works for about 1 out 4 events. I do see the events going to the Logger Sink so I know the Source is doing its job. However, for the custom Sink I simply get Event is null messages. I tried adjusting the channel capacity from 10 to 1 but no difference.
>  
>  
> Thanks.
>  
>  
> Here’s my conf file
>  
>  
> # flume-httpxmlhttp.conf: A single-node Flume with Http Source and Http sink configuration
>  
> # Name the components on this agent
> agent1.sources = r1
> agent1.channels = c1
>  
> # Describe/configure the source
> agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
> agent1.sources.r1.port = 5140
> agent1.sources.r1.handler = main.java.org.apache.flume.source.http.XMLHandler
> agent1.sources.r1.handler.nickname = random props
>  
> # Setup the sinks
> agent1.sinks = httpsink logsink
>  
> # Describe the sink
> agent1.sinks.logsink.type = logger
>  
> # Describe the sink
> agent1.sinks.httpsink.type = main.java.org.apache.flume.sink.http.HttpPostSink
> agent1.sinks.httpsink.serverAddress = <my http server URL>
>  
> # Use a channel which buffers events in memory
> agent1.channels.c1.type = memory
> agent1.channels.c1.capacity = 1
> agent1.channels.c1.transactionCapacity = 1
>  
> # Bind the source and sink to the channel
> agent1.sources.r1.channels = c1
> agent1.sinks.logsink.channel = c1
> agent1.sinks.httpsink.channel = c1
>  
>  
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB