Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume service stopped automatically


Copy link to this message
-
Re: Flume service stopped automatically
Weird... It could have been the linux oom killer? You'd see something in
/var/log/message if that was the case.
On Sat, Jun 1, 2013 at 2:47 AM, Lenin Raj <[EMAIL PROTECTED]> wrote:

> Hello,
>
> I have a flume service which pulls twitter data and sinks to HDFS. I
> started it last night at 8 PM. It was going well. When I checked the status
> today morning, it seems flume service stopped at around 11:15 PM.
>
> I could not find any valid reason in the log file. Flume is doing "Flume
> node stopping - TwitterAgent" automatically without throwing any error.
>
>
> 31 May 2013 23:13:32,226 INFO  [hdfs-HDFS-call-runner-7]
> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208)  - Creating
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414624.tmp
> 31 May 2013 23:14:02,247 INFO  [hdfs-HDFS-roll-timer-0]
> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427)  - Renaming
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624.tmp
> to
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624
> 31 May 2013 23:14:26,260 INFO  [hdfs-HDFS-call-runner-1]
> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208)  - Creating
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414625.tmp
> 31 May 2013 23:14:56,288 INFO  [hdfs-HDFS-roll-timer-0]
> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427)  - Renaming
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625.tmp
> to
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625
> 31 May 2013 23:15:02,317 INFO  [hdfs-HDFS-call-runner-5]
> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208)  - Creating
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414626.tmp
> 31 May 2013 23:15:32,340 INFO  [hdfs-HDFS-roll-timer-0]
> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427)  - Renaming
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626.tmp
> to
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626
> 31 May 2013 23:15:43,632 INFO  [hdfs-HDFS-call-runner-9]
> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208)  - Creating
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414627.tmp
> 31 May 2013 23:15:56,604 INFO  [node-shutdownHook]
> (org.apache.flume.node.FlumeNode.stop:67)  - Flume node stopping -
> TwitterAgent
> 31 May 2013 23:15:56,604 INFO  [node-shutdownHook]
> (org.apache.flume.lifecycle.LifecycleSupervisor.stop:78)  - Stopping
> lifecycle supervisor 7
> 31 May 2013 23:15:56,606 INFO  [node-shutdownHook]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stop:215)  -
> Node manager stopping
> 31 May 2013 23:15:56,606 INFO  [node-shutdownHook]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:68)
> - Shutting down configuration: {
> sourceRunners:{Twitter=EventDrivenSourceRunner: {
> source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} }}
> sinkRunners:{HDFS=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@1d80e6d counterGroup:{
> name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=1252}
>  } }} channels:{MemChannel=org.apache.flume.channel.MemoryChannel{name:
> MemChannel}} }
> 31 May 2013 23:15:56,607 INFO  [node-shutdownHook]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:72)
> - Stopping Source Twitter
> 31 May 2013 23:15:56,607 INFO  [node-shutdownHook]
> (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:156)  -
> Stopping component: EventDrivenSourceRunner: {
> source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} }
> 31 May 2013 23:15:56,682 WARN  [Twitter Stream consumer-1[Disposing
> thread]] (twitter4j.internal.logging.SLF4JLogger.warn:99)  - Stream already
> closed.
> 31 May 2013 23:15:56,682 INFO  [node-shutdownHook]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:82)

Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB