Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> ERROR connector.DirectDriver: Exiting driver logicalNode agent2-18 in error state TailSource | fail because Attempting to open a stub sink 'fail'!


Copy link to this message
-
Re: ERROR connector.DirectDriver: Exiting driver logicalNode agent2-18 in error state TailSource | fail because Attempting to open a stub sink 'fail'!
Hi Andy,

How about trying "refreshAll" command to see it helps?

It seems that logicalSource and logicalSink are not working perfectly
especially in finding and mapping logical node to physical node during
translation process in master node. I think "map Hadoop48 collect1"
should be done earlier so that master has mapping information for
collector1 to Hadoop48(physical node) so that translation can be done
properly when related configs were created.

for example,
(config) collect1: null | null

map Hadoop48 collect1

(config) collect1: logicalSource |
collectorSink("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-",5000,seqfile);

(config) agent1: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
(config) agent2: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");

map Hadoop46 agent1
map Hadoop47 agent2

and, there seems typo in node name; you mapped "aggent1" to Hadoop46 but
name of the config you had created was "agent1"; it is why aggent1 on
Hadoop46 is "IDLE".

- JS

On 1/23/13 6:21 PM, ??? wrote:
> hello,
> I add logical config to flume:
> agent1: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
> agent2: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
> collect1: logicalSource |
> collectorSink("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-",5000,seqfile);
>
> map Hadoop46 agent1
> map Hadoop47 agent2
> map Hadoop48 collect1
>
> ===========================> I run master on Hadoop48,
> it reports:
> 2013-01-23 16:49:27,153 [main] INFO rolling.RollSink: Created
> RollSink: trigger=[TimeTrigger: maxAge=5000
> tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@7a960b08]
> checkPeriodMs = 250
> spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:27,158 [main] INFO rolling.RollSink: Created
> RollSink: trigger=[TimeTrigger: maxAge=5000
> tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@1d29109e]
> checkPeriodMs = 250
> spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:27,207 [main] INFO master.ZooKeeperConfigStore:
> Created new config at /flume-cfgs/cfg-0000000104
> 2013-01-23 16:49:27,207 [main-EventThread] INFO
> master.ZooKeeperConfigStore: Config was updated - reloading
> 2013-01-23 16:49:27,255 [main] INFO master.ZooKeeperConfigStore:
> Created new config at /flume-cfgs/cfg-0000000105
> 2013-01-23 16:49:27,255 [main-EventThread] INFO
> master.ZooKeeperConfigStore: Config was updated - reloading
> 2013-01-23 16:49:27,307 [main] WARN
> logical.LogicalConfigurationManager: Logical node 'collect1' has not
> been assigned to a physical node yet
>
> ================> run agent1 on Hadoop46, it reports:
>
> 2013-01-23 16:49:45,070 [Heartbeat] INFO rolling.RollSink: Created
> RollSink: trigger=[TimeTrigger: maxAge=5000
> tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@1fd54f77]
> checkPeriodMs = 250
> spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:45,073 [Heartbeat] INFO agent.LogicalNodeManager:
> creating new logical node collect1
> 2013-01-23 16:49:45,076 [logicalNode collect1-18] ERROR
> connector.DirectDriver: Closing down due to exception on open calls
> 2013-01-23 16:49:45,076 [logicalNode collect1-18] INFO
> connector.DirectDriver: Connector logicalNode collect1-18 exited with
> error: Attempting to open a Stub Source!
> 2013-01-23 16:49:45,077 [logicalNode collect1-18] ERROR
> connector.DirectDriver: Error closing logicalNode collect1-18 source:
> Attempting to close a Stub Source!
> 2013-01-23 16:49:45,077 [logicalNode collect1-18] INFO
> rolling.RollSink: closing RollSink
> 'escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:45,081 [logicalNode collect1-18] INFO
> rolling.RollSink: double close
> 'escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:45,081 [logicalNode collect1-18] ERROR
Jeong-shik Jang / [EMAIL PROTECTED]
Gruter, Inc., R&D Team Leader
www.gruter.com
Enjoy Connecting
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB