Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Flume service stopped automatically


+
Lenin Raj 2013-06-01, 07:47
Copy link to this message
-
Re: Flume service stopped automatically
Weird... It could have been the linux oom killer? You'd see something in
/var/log/message if that was the case.
On Sat, Jun 1, 2013 at 2:47 AM, Lenin Raj <[EMAIL PROTECTED]> wrote:

> Hello,
>
> I have a flume service which pulls twitter data and sinks to HDFS. I
> started it last night at 8 PM. It was going well. When I checked the status
> today morning, it seems flume service stopped at around 11:15 PM.
>
> I could not find any valid reason in the log file. Flume is doing "Flume
> node stopping - TwitterAgent" automatically without throwing any error.
>
>
> 31 May 2013 23:13:32,226 INFO  [hdfs-HDFS-call-runner-7]
> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208)  - Creating
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414624.tmp
> 31 May 2013 23:14:02,247 INFO  [hdfs-HDFS-roll-timer-0]
> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427)  - Renaming
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624.tmp
> to
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624
> 31 May 2013 23:14:26,260 INFO  [hdfs-HDFS-call-runner-1]
> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208)  - Creating
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414625.tmp
> 31 May 2013 23:14:56,288 INFO  [hdfs-HDFS-roll-timer-0]
> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427)  - Renaming
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625.tmp
> to
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625
> 31 May 2013 23:15:02,317 INFO  [hdfs-HDFS-call-runner-5]
> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208)  - Creating
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414626.tmp
> 31 May 2013 23:15:32,340 INFO  [hdfs-HDFS-roll-timer-0]
> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427)  - Renaming
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626.tmp
> to
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626
> 31 May 2013 23:15:43,632 INFO  [hdfs-HDFS-call-runner-9]
> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208)  - Creating
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414627.tmp
> 31 May 2013 23:15:56,604 INFO  [node-shutdownHook]
> (org.apache.flume.node.FlumeNode.stop:67)  - Flume node stopping -
> TwitterAgent
> 31 May 2013 23:15:56,604 INFO  [node-shutdownHook]
> (org.apache.flume.lifecycle.LifecycleSupervisor.stop:78)  - Stopping
> lifecycle supervisor 7
> 31 May 2013 23:15:56,606 INFO  [node-shutdownHook]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stop:215)  -
> Node manager stopping
> 31 May 2013 23:15:56,606 INFO  [node-shutdownHook]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:68)
> - Shutting down configuration: {
> sourceRunners:{Twitter=EventDrivenSourceRunner: {
> source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} }}
> sinkRunners:{HDFS=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@1d80e6d counterGroup:{
> name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=1252}
>  } }} channels:{MemChannel=org.apache.flume.channel.MemoryChannel{name:
> MemChannel}} }
> 31 May 2013 23:15:56,607 INFO  [node-shutdownHook]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:72)
> - Stopping Source Twitter
> 31 May 2013 23:15:56,607 INFO  [node-shutdownHook]
> (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:156)  -
> Stopping component: EventDrivenSourceRunner: {
> source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} }
> 31 May 2013 23:15:56,682 WARN  [Twitter Stream consumer-1[Disposing
> thread]] (twitter4j.internal.logging.SLF4JLogger.warn:99)  - Stream already
> closed.
> 31 May 2013 23:15:56,682 INFO  [node-shutdownHook]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:82)

Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
+
Lenin Raj 2013-06-01, 15:49
+
Alexander Alten-Lorenz 2013-06-03, 09:44