Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> flume master down, flume node still has multiple orphan processes


+
Shengjie Min 2012-12-19, 11:34
+
Nitin Pawar 2012-12-19, 11:54
+
Shengjie Min 2012-12-19, 12:08
Copy link to this message
-
Re: flume master down, flume node still has multiple orphan processes
You have to kill -9 them, when you let run a jstack on a JVM pid you'll see WAITING threads.

Best,
 Alex
On Dec 19, 2012, at 12:34 PM, Shengjie Min <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I noticed that when my flume master is down for a quite while. My flume
> agent is retrying to connect to the master, i can tell that from the flume
> logs as below;
>
> 2012-12-19 11:11:42,126 WARN com.cloudera.flume.agent.MultiMasterRPC: Could
> not connect to any master nodes (tried 1: [flume-master-01:35872])
> 2012-12-19 11:11:52,140 WARN com.cloudera.flume.agent.MultiMasterRPC: Could
> not connect to any master nodes (tried 1: [flume-master-01:35872])
> 2012-12-19 11:12:02,152 WARN com.cloudera.flume.agent.MultiMasterRPC: Could
> not connect to any master nodes (tried 1: [flume-master-01:35872])
> 2012-12-19 11:12:12,162 WARN com.cloudera.flume.agent.MultiMasterRPC: Could
> not connect to any master nodes (tried 1: [flume-master-01:35872])
>
> from "top", I've seen a lot of java processes owned by flume. followed my
> instinct, I stopped flume agent, after a while, I run "top" on it, I still
> have a lot of processes owned by flume user.
>
> 12735 flume     20   0 1487m  26m  128 S  0.3  1.3   0:05.41 java
>
>
>
>
> 13828 flume     20   0 1487m   24   24 S  0.3  0.0   8:11.92 java
>
>
>
>
> 14442 flume     20   0 1487m    0    0 S  0.3  0.0   8:06.81 java
>
>
>
>
> 15033 flume     20   0 1487m  26m  164 S  0.3  1.3   0:04.29 java
>
>
>
>
> 15453 flume     20   0 1487m    4    4 S  0.3  0.0   8:07.55 java
>
>
>
>
> 15964 flume     20   0 1487m  27m  164 S  0.3  1.4   0:04.09 java
>
>
>
>
> 16098 flume     20   0 1487m    0    0 S  0.3  0.0   8:12.67 java
> ......................
>
> Why so many processes are generated here and they don't go away even when
> flume is stopped.
>
> --
> All the best,
> Shengjie Min
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB