Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> ERROR connector.DirectDriver: Exiting driver logicalNode agent2-18 in error state TailSource | fail because Attempting to open a stub sink 'fail'!


Copy link to this message
-
ERROR connector.DirectDriver: Exiting driver logicalNode agent2-18 in error state TailSource | fail because Attempting to open a stub sink 'fail'!
hello,
I add logical config to flume:
agent1: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
agent2: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
collect1: logicalSource |
collectorSink("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-",5000,seqfile);

map Hadoop46 agent1
map Hadoop47 agent2
map Hadoop48 collect1

===========================I run master on Hadoop48,
it reports:
2013-01-23 16:49:27,153 [main] INFO rolling.RollSink: Created RollSink:
trigger=[TimeTrigger: maxAge=5000
tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@7a960b08]
checkPeriodMs = 250
spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'
2013-01-23 16:49:27,158 [main] INFO rolling.RollSink: Created RollSink:
trigger=[TimeTrigger: maxAge=5000
tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@1d29109e]
checkPeriodMs = 250
spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'
2013-01-23 16:49:27,207 [main] INFO master.ZooKeeperConfigStore: Created
new config at /flume-cfgs/cfg-0000000104
2013-01-23 16:49:27,207 [main-EventThread] INFO
master.ZooKeeperConfigStore: Config was updated - reloading
2013-01-23 16:49:27,255 [main] INFO master.ZooKeeperConfigStore: Created
new config at /flume-cfgs/cfg-0000000105
2013-01-23 16:49:27,255 [main-EventThread] INFO
master.ZooKeeperConfigStore: Config was updated - reloading
2013-01-23 16:49:27,307 [main] WARN logical.LogicalConfigurationManager:
Logical node 'collect1' has not been assigned to a physical node yet

================run agent1 on Hadoop46, it reports:

2013-01-23 16:49:45,070 [Heartbeat] INFO rolling.RollSink: Created
RollSink: trigger=[TimeTrigger: maxAge=5000
tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@1fd54f77]
checkPeriodMs = 250
spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'
2013-01-23 16:49:45,073 [Heartbeat] INFO agent.LogicalNodeManager: creating
new logical node collect1
2013-01-23 16:49:45,076 [logicalNode collect1-18] ERROR
connector.DirectDriver: Closing down due to exception on open calls
2013-01-23 16:49:45,076 [logicalNode collect1-18] INFO
connector.DirectDriver: Connector logicalNode collect1-18 exited with
error: Attempting to open a Stub Source!
2013-01-23 16:49:45,077 [logicalNode collect1-18] ERROR
connector.DirectDriver: Error closing logicalNode collect1-18 source:
Attempting to close a Stub Source!
2013-01-23 16:49:45,077 [logicalNode collect1-18] INFO rolling.RollSink:
closing RollSink
'escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'
2013-01-23 16:49:45,081 [logicalNode collect1-18] INFO rolling.RollSink:
double close
'escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'
2013-01-23 16:49:45,081 [logicalNode collect1-18] ERROR
connector.DirectDriver: Exiting driver logicalNode collect1-18 in error
state StubSource | Collector because Attempting to open a Stub Source!
2013-01-23 16:49:45,091 [Check config] INFO rolling.RollSink: Created
RollSink: trigger=[TimeTrigger: maxAge=5000
tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@5780208f]
checkPeriodMs = 250
spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'

==============and agent2 on Hadoop47 , also logs error.
[zhouhh@Hadoop47 ~]$ flume node

2013-01-23 17:11:38,454 [main] INFO mortbay.log: Started
SelectChannelConnector@0.0.0.0:35862
2013-01-23 17:11:38,454 [main] INFO util.InternalHttpServer: Server started
2013-01-23 17:11:38,468 [Heartbeat] INFO agent.MultiMasterRPC: No active
master RPC connection
2013-01-23 17:11:38,495 [Heartbeat] INFO agent.ThriftMasterRPC: Connected
to master at hadoop48:35872
2013-01-23 17:11:38,687 [main] INFO agent.LogicalNodeManager: creating new
logical node Hadoop47
2013-01-23 17:11:38,819 [main] WARN agent.FlumeNode: Flume is using Hadoop
core 1.0.4 which does not support Security / Authentication: null
2013-01-23 17:11:43,529 [Heartbeat] INFO agent.LogicalNodeManager: creating
new logical node agent2
2013-01-23 17:11:43,536 [logicalNode agent2-18] ERROR
connector.DirectDriver: Closing down due to exception on open calls
2013-01-23 17:11:43,537 [logicalNode agent2-18] INFO
connector.DirectDriver: Connector logicalNode agent2-18 exited with error:
Attempting to open a stub sink 'fail'!
2013-01-23 17:11:43,537 [logicalNode agent2-18] ERROR
connector.DirectDriver: Exiting driver logicalNode agent2-18 in error state
TailSource | fail because Attempting to open a stub sink 'fail'!
2013-01-23 17:11:43,540 [TailThread-0] INFO text.TailSource: TailThread has
exited
2013-01-23 17:11:43,553 [Check config] INFO agent.LogicalNode: Node config
successfully set to com.cloudera.flume.conf.FlumeConfigData@76a0f9cb
2013-01-23 17:11:43,554 [logicalNode agent2-20] ERROR
connector.DirectDriver: Closing down due to exception on open calls
2013-01-23 17:11:43,554 [logicalNode agent2-20] INFO
connector.DirectDriver: Connector logicalNode agent2-20 exited with error:
Attempting to open a stub sink 'fail'!
2013-01-23 17:11:43,554 [logicalNode agent2-20] ERROR
connector.DirectDriver: Exiting driver logicalNode agent2-20 in error state
TailSource | fail because Attempting to open a stub sink 'fail'!
2013-01-23 17:11:43,554 [TailThread-1] INFO text.TailSource: TailThread has
exited

http://hadoop48:35871/flumemaster.jsp
Node status logical node physical nodehost name statusversionlast seen
delta (s) last seenHadoop46 Hadoop46Hadoop46IDLEnone 2Wed Jan 23 17:07:05
CST 2013 Hadoop47Hadoop47Hadoop47 IDLEnone1Wed Jan 23 17:07:06 CST 2013
Hadoop48Hadoop48Hadoop48 IDLEWed Jan 23 16:49:27 CST 2013 0Wed Jan 23
17:07:08 CST 2013WX-wuwb.TKOffice.local WX-wuwb.TKOffice.local
WX-wuwb.TKOffice.localERROR Wed Jan 23 16:49:27 CST 20130Wed Jan 23
17:07:07 CST 2013 agent2Hadoop47Hadoop47ERROR Wed Jan 23 16:49:27 CST
20131Wed Jan 23 17:07:06 CST 2013
aggent1Hadoop46 Hadoop46IDLEnone2 Wed Jan 23 17:07:05 CST 2013co1co1Hadoop48
LOSTWed Jan 23 09:56:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB