Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> ERROR connector.DirectDriver: Exiting driver logicalNode agent2-18 in error state TailSource | fail because Attempting to open a stub sink 'fail'!


+
周梦想 2013-01-23, 09:21
+
Jeong-shik Jang 2013-01-23, 14:36
Copy link to this message
-
Re: ERROR connector.DirectDriver: Exiting driver logicalNode agent2-18 in error state TailSource | fail because Attempting to open a stub sink 'fail'!
thank you jeong-shik, I will try this:)
Andy

2013/1/23 Jeong-shik Jang <[EMAIL PROTECTED]>

>  Hi Andy,
>
> How about trying "refreshAll" command to see it helps?
>
> It seems that logicalSource and logicalSink are not working perfectly
> especially in finding and mapping logical node to physical node during
> translation process in master node. I think "map Hadoop48 collect1" should
> be done earlier so that master has mapping information for collector1 to
> Hadoop48(physical node) so that translation can be done properly when
> related configs were created.
>
> for example,
> (config) collect1: null | null
>
> map Hadoop48 collect1
>
> (config) collect1: logicalSource |
> collectorSink("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-",5000,seqfile);
>
> (config) agent1: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
> (config) agent2: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
>
> map Hadoop46 agent1
>  map Hadoop47 agent2
>
> and, there seems typo in node name; you mapped "aggent1" to Hadoop46 but
> name of the config you had created was "agent1"; it is why aggent1 on
> Hadoop46 is "IDLE".
>
> - JS
>
> On 1/23/13 6:21 PM, 周梦想 wrote:
>
> hello,
> I add logical config to flume:
> agent1: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
> agent2: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
> collect1: logicalSource |
> collectorSink("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-",5000,seqfile);
>
>  map Hadoop46 agent1
>  map Hadoop47 agent2
>  map Hadoop48 collect1
>
>  ===========================> I run master on Hadoop48,
> it reports:
>  2013-01-23 16:49:27,153 [main] INFO rolling.RollSink: Created RollSink:
> trigger=[TimeTrigger: maxAge=5000
> tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@7a960b08]
> checkPeriodMs = 250
> spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:27,158 [main] INFO rolling.RollSink: Created RollSink:
> trigger=[TimeTrigger: maxAge=5000
> tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@1d29109e]
> checkPeriodMs = 250
> spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:27,207 [main] INFO master.ZooKeeperConfigStore: Created
> new config at /flume-cfgs/cfg-0000000104
> 2013-01-23 16:49:27,207 [main-EventThread] INFO
> master.ZooKeeperConfigStore: Config was updated - reloading
> 2013-01-23 16:49:27,255 [main] INFO master.ZooKeeperConfigStore: Created
> new config at /flume-cfgs/cfg-0000000105
> 2013-01-23 16:49:27,255 [main-EventThread] INFO
> master.ZooKeeperConfigStore: Config was updated - reloading
> 2013-01-23 16:49:27,307 [main] WARN logical.LogicalConfigurationManager:
> Logical node 'collect1' has not been assigned to a physical node yet
>
>  ================> run agent1 on Hadoop46, it reports:
>
>  2013-01-23 16:49:45,070 [Heartbeat] INFO rolling.RollSink: Created
> RollSink: trigger=[TimeTrigger: maxAge=5000
> tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@1fd54f77]
> checkPeriodMs = 250
> spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:45,073 [Heartbeat] INFO agent.LogicalNodeManager:
> creating new logical node collect1
> 2013-01-23 16:49:45,076 [logicalNode collect1-18] ERROR
> connector.DirectDriver: Closing down due to exception on open calls
> 2013-01-23 16:49:45,076 [logicalNode collect1-18] INFO
> connector.DirectDriver: Connector logicalNode collect1-18 exited with
> error: Attempting to open a Stub Source!
> 2013-01-23 16:49:45,077 [logicalNode collect1-18] ERROR
> connector.DirectDriver: Error closing logicalNode collect1-18 source:
> Attempting to close a Stub Source!
> 2013-01-23 16:49:45,077 [logicalNode collect1-18] INFO rolling.RollSink:
> closing RollSink
> 'escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:45,081 [logicalNode collect1-18] INFO rolling.RollSink:
> double close