Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Flume Ng replaying events when the source is idle


Copy link to this message
-
Re: Flume Ng replaying events when the source is idle
Sagar Mehta 2013-03-04, 23:06
I think we found the issue, not sure if this is the root cause but looks
highly correlated.

So we manage configs using puppet which currently runs in a cron mode with
following configuration

## puppetrun Cron Job
20,50 * * * *  root sleep $((RANDOM\%60)) > /dev/null 2>&1; puppet agent
--onetime --no-daemonize --logdest syslog > /dev/null 2>&1

*Note - the times at which puppet is run along with the time-stamps in the
listing below.*

Also after combing through flume logs, we noticed Flume is reloading the
configuration after every puppet run

sagar@drspock ~/temp $ cat flume.log.2013-03-03 | egrep -i "reloading" |
head -5
2013-03-03 00:20:44,174 [conf-file-poller-0] INFO
 org.apache.flume.conf.properties.PropertiesFileConfigurationProvider -
Reloading configuration file:/opt/flume/conf/hdfs.conf
2013-03-03 00:51:14,374 [conf-file-poller-0] INFO
 org.apache.flume.conf.properties.PropertiesFileConfigurationProvider -
Reloading configuration file:/opt/flume/conf/hdfs.conf
2013-03-03 01:21:15,072 [conf-file-poller-0] INFO
 org.apache.flume.conf.properties.PropertiesFileConfigurationProvider -
Reloading configuration file:/opt/flume/conf/hdfs.conf
2013-03-03 01:51:15,778 [conf-file-poller-0] INFO
 org.apache.flume.conf.properties.PropertiesFileConfigurationProvider -
Reloading configuration file:/opt/flume/conf/hdfs.conf
2013-03-03 02:20:46,481 [conf-file-poller-0] INFO
 org.apache.flume.conf.properties.PropertiesFileConfigurationProvider -
Reloading configuration file:/opt/flume/conf/hdfs.conf

The way we have our current setup, the flume config file
namely /opt/flume/conf/hdfs.conf is re-written after every puppet run due
to variable interpolation in the template.

*We are still not sure what is causing Flume to reload the config file, and
even if the file is reloaded why are the same events getting replayed [the
state should be saved somewhere on disk - thats what the file channel is
for I thought]*

Any pointers/insights appreciated.

Sagar
On Mon, Mar 4, 2013 at 2:42 PM, Sagar Mehta <[EMAIL PROTECTED]> wrote:

> Guys,
>
> Yes this issue was also seen in the memory channel. In fact when we moved
> to File based channel, we initially thought  this issue won't occur since
> it stores check points.
>
> Anyways below are all files for collector110 [whose source didn't receive
> any events] and you can see all the replays below. I have attached the
> corresponding flume log file for the same day.
>
> hadoop@jobtracker301:/home/smehta$ hls
> /ngpipes-raw-logs/2013-03-03/*/collector110* |  head -5
> -rw-r--r--   3 hadoop supergroup       1594 2013-03-03 00:20
> /ngpipes-raw-logs/2013-03-03/0000/collector110.ngpipes.sac.ngmoco.com.1362270044367.gz
> -rw-r--r--   3 hadoop supergroup       1594 2013-03-03 00:51
> /ngpipes-raw-logs/2013-03-03/0000/collector110.ngpipes.sac.ngmoco.com.1362271875065.gz
> -rw-r--r--   3 hadoop supergroup       1594 2013-03-03 01:21
> /ngpipes-raw-logs/2013-03-03/0100/collector110.ngpipes.sac.ngmoco.com.1362273675770.gz
> -rw-r--r--   3 hadoop supergroup       1594 2013-03-03 01:51
> /ngpipes-raw-logs/2013-03-03/0100/collector110.ngpipes.sac.ngmoco.com.1362275476474.gz
> -rw-r--r--   3 hadoop supergroup       1594 2013-03-03 02:20
> /ngpipes-raw-logs/2013-03-03/0200/collector110.ngpipes.sac.ngmoco.com.1362277246704.gz
>
> Also in the attached flume log, you can see the replay stuff I'm talking
> about - Please note the source received no events during this time.
>
> sagar@drspock ~/temp $ cat flume.log.2013-03-03 | egrep -i "Queue Size
> after replay" | head
> 2013-03-03 00:20:44,355 [lifecycleSupervisor-1-3] INFO
>  org.apache.flume.channel.file.FileChannel - Queue Size after replay: 10
> [channel=channel1]
> 2013-03-03 00:20:44,356 [lifecycleSupervisor-1-4] INFO
>  org.apache.flume.channel.file.FileChannel - Queue Size after replay: 10
> [channel=channel2]
> 2013-03-03 00:51:14,571 [lifecycleSupervisor-1-7] INFO
>  org.apache.flume.channel.file.FileChannel - Queue Size after replay: 0
> [channel=channel2]