Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> ERROR connector.DirectDriver: Exiting driver logicalNode agent2-18 in error state TailSource | fail because Attempting to open a stub sink 'fail'!


+
周梦想 2013-01-23, 09:21
+
Jeong-shik Jang 2013-01-23, 14:36
Copy link to this message
-
Re: ERROR connector.DirectDriver: Exiting driver logicalNode agent2-18 in error state TailSource | fail because Attempting to open a stub sink 'fail'!
thank you jeong-shik, I will try this:)
Andy

2013/1/23 Jeong-shik Jang <[EMAIL PROTECTED]>

>  Hi Andy,
>
> How about trying "refreshAll" command to see it helps?
>
> It seems that logicalSource and logicalSink are not working perfectly
> especially in finding and mapping logical node to physical node during
> translation process in master node. I think "map Hadoop48 collect1" should
> be done earlier so that master has mapping information for collector1 to
> Hadoop48(physical node) so that translation can be done properly when
> related configs were created.
>
> for example,
> (config) collect1: null | null
>
> map Hadoop48 collect1
>
> (config) collect1: logicalSource |
> collectorSink("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-",5000,seqfile);
>
> (config) agent1: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
> (config) agent2: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
>
> map Hadoop46 agent1
>  map Hadoop47 agent2
>
> and, there seems typo in node name; you mapped "aggent1" to Hadoop46 but
> name of the config you had created was "agent1"; it is why aggent1 on
> Hadoop46 is "IDLE".
>
> - JS
>
> On 1/23/13 6:21 PM, 周梦想 wrote:
>
> hello,
> I add logical config to flume:
> agent1: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
> agent2: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
> collect1: logicalSource |
> collectorSink("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-",5000,seqfile);
>
>  map Hadoop46 agent1
>  map Hadoop47 agent2
>  map Hadoop48 collect1
>
>  ===========================> I run master on Hadoop48,
> it reports:
>  2013-01-23 16:49:27,153 [main] INFO rolling.RollSink: Created RollSink:
> trigger=[TimeTrigger: maxAge=5000
> tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@7a960b08]
> checkPeriodMs = 250
> spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:27,158 [main] INFO rolling.RollSink: Created RollSink:
> trigger=[TimeTrigger: maxAge=5000
> tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@1d29109e]
> checkPeriodMs = 250
> spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:27,207 [main] INFO master.ZooKeeperConfigStore: Created
> new config at /flume-cfgs/cfg-0000000104
> 2013-01-23 16:49:27,207 [main-EventThread] INFO
> master.ZooKeeperConfigStore: Config was updated - reloading
> 2013-01-23 16:49:27,255 [main] INFO master.ZooKeeperConfigStore: Created
> new config at /flume-cfgs/cfg-0000000105
> 2013-01-23 16:49:27,255 [main-EventThread] INFO
> master.ZooKeeperConfigStore: Config was updated - reloading
> 2013-01-23 16:49:27,307 [main] WARN logical.LogicalConfigurationManager:
> Logical node 'collect1' has not been assigned to a physical node yet
>
>  ================> run agent1 on Hadoop46, it reports:
>
>  2013-01-23 16:49:45,070 [Heartbeat] INFO rolling.RollSink: Created
> RollSink: trigger=[TimeTrigger: maxAge=5000
> tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@1fd54f77]
> checkPeriodMs = 250
> spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:45,073 [Heartbeat] INFO agent.LogicalNodeManager:
> creating new logical node collect1
> 2013-01-23 16:49:45,076 [logicalNode collect1-18] ERROR
> connector.DirectDriver: Closing down due to exception on open calls
> 2013-01-23 16:49:45,076 [logicalNode collect1-18] INFO
> connector.DirectDriver: Connector logicalNode collect1-18 exited with
> error: Attempting to open a Stub Source!
> 2013-01-23 16:49:45,077 [logicalNode collect1-18] ERROR
> connector.DirectDriver: Error closing logicalNode collect1-18 source:
> Attempting to close a Stub Source!
> 2013-01-23 16:49:45,077 [logicalNode collect1-18] INFO rolling.RollSink:
> closing RollSink
> 'escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:45,081 [logicalNode collect1-18] INFO rolling.RollSink:
> double close
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB