Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Dropped events


+
Vikram Kulkarni 2013-03-12, 18:03
+
Hari Shreedharan 2013-03-12, 18:05
Copy link to this message
-
RE: Dropped events
I had it at 10 originally and the capacity at 100.
Yes, I am committing the transaction and closing it (and backoff) even if the Event is null.

Thanks,
Vikram

From: Hari Shreedharan [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, March 12, 2013 11:06 AM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Dropped events

Your channel capacity is set to 1. Are you sure you really want that? If a null event is returned by the channel, then you should commit and close the transaction, optionally backoff (usually, this is when a batch was empty), and then try again. You should probably also use a higher transaction capacity.
Hari

--
Hari Shreedharan
On Tuesday, March 12, 2013 at 11:03 AM, Vikram Kulkarni wrote:

I have my custom Source and Sink that I have hooked with a memory channel but I am noticing that it is not very consistent. Even after sending many events to the Source the Sink’s event is still null. It works for about 1 out 4 events. I do see the events going to the Logger Sink so I know the Source is doing its job. However, for the custom Sink I simply get Event is null messages. I tried adjusting the channel capacity from 10 to 1 but no difference.

Thanks.

Here’s my conf file
# flume-httpxmlhttp.conf: A single-node Flume with Http Source and Http sink configuration

# Name the components on this agent
agent1.sources = r1
agent1.channels = c1

# Describe/configure the source
agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
agent1.sources.r1.port = 5140
agent1.sources.r1.handler = main.java.org.apache.flume.source.http.XMLHandler
agent1.sources.r1.handler.nickname = random props

# Setup the sinks
agent1.sinks = httpsink logsink

# Describe the sink
agent1.sinks.logsink.type = logger

# Describe the sink
agent1.sinks.httpsink.type = main.java.org.apache.flume.sink.http.HttpPostSink
agent1.sinks.httpsink.serverAddress = <my http server URL>

# Use a channel which buffers events in memory
agent1.channels.c1.type = memory
agent1.channels.c1.capacity = 1
agent1.channels.c1.transactionCapacity = 1

# Bind the source and sink to the channel
agent1.sources.r1.channels = c1
agent1.sinks.logsink.channel = c1
agent1.sinks.httpsink.channel = c1

+
Vikram Kulkarni 2013-03-13, 18:36
+
Connor Woodson 2013-03-15, 19:22
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB