Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - ERROR connector.DirectDriver: Exiting driver logicalNode agent2-18 in error state TailSource | fail because Attempting to open a stub sink 'fail'!


Copy link to this message
-
ERROR connector.DirectDriver: Exiting driver logicalNode agent2-18 in error state TailSource | fail because Attempting to open a stub sink 'fail'!
周梦想 2013-01-23, 09:21
hello,
I add logical config to flume:
agent1: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
agent2: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
collect1: logicalSource |
collectorSink("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-",5000,seqfile);

map Hadoop46 agent1
map Hadoop47 agent2
map Hadoop48 collect1

===========================I run master on Hadoop48,
it reports:
2013-01-23 16:49:27,153 [main] INFO rolling.RollSink: Created RollSink:
trigger=[TimeTrigger: maxAge=5000
tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@7a960b08]
checkPeriodMs = 250
spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'
2013-01-23 16:49:27,158 [main] INFO rolling.RollSink: Created RollSink:
trigger=[TimeTrigger: maxAge=5000
tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@1d29109e]
checkPeriodMs = 250
spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'
2013-01-23 16:49:27,207 [main] INFO master.ZooKeeperConfigStore: Created
new config at /flume-cfgs/cfg-0000000104
2013-01-23 16:49:27,207 [main-EventThread] INFO
master.ZooKeeperConfigStore: Config was updated - reloading
2013-01-23 16:49:27,255 [main] INFO master.ZooKeeperConfigStore: Created
new config at /flume-cfgs/cfg-0000000105
2013-01-23 16:49:27,255 [main-EventThread] INFO
master.ZooKeeperConfigStore: Config was updated - reloading
2013-01-23 16:49:27,307 [main] WARN logical.LogicalConfigurationManager:
Logical node 'collect1' has not been assigned to a physical node yet

================run agent1 on Hadoop46, it reports:

2013-01-23 16:49:45,070 [Heartbeat] INFO rolling.RollSink: Created
RollSink: trigger=[TimeTrigger: maxAge=5000
tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@1fd54f77]
checkPeriodMs = 250
spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'
2013-01-23 16:49:45,073 [Heartbeat] INFO agent.LogicalNodeManager: creating
new logical node collect1
2013-01-23 16:49:45,076 [logicalNode collect1-18] ERROR
connector.DirectDriver: Closing down due to exception on open calls
2013-01-23 16:49:45,076 [logicalNode collect1-18] INFO
connector.DirectDriver: Connector logicalNode collect1-18 exited with
error: Attempting to open a Stub Source!
2013-01-23 16:49:45,077 [logicalNode collect1-18] ERROR
connector.DirectDriver: Error closing logicalNode collect1-18 source:
Attempting to close a Stub Source!
2013-01-23 16:49:45,077 [logicalNode collect1-18] INFO rolling.RollSink:
closing RollSink
'escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'
2013-01-23 16:49:45,081 [logicalNode collect1-18] INFO rolling.RollSink:
double close
'escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'
2013-01-23 16:49:45,081 [logicalNode collect1-18] ERROR
connector.DirectDriver: Exiting driver logicalNode collect1-18 in error
state StubSource | Collector because Attempting to open a Stub Source!
2013-01-23 16:49:45,091 [Check config] INFO rolling.RollSink: Created
RollSink: trigger=[TimeTrigger: maxAge=5000
tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@5780208f]
checkPeriodMs = 250
spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'

==============and agent2 on Hadoop47 , also logs error.
[zhouhh@Hadoop47 ~]$ flume node

2013-01-23 17:11:38,454 [main] INFO mortbay.log: Started
SelectChannelConnector@0.0.0.0:35862
2013-01-23 17:11:38,454 [main] INFO util.InternalHttpServer: Server started
2013-01-23 17:11:38,468 [Heartbeat] INFO agent.MultiMasterRPC: No active
master RPC connection
2013-01-23 17:11:38,495 [Heartbeat] INFO agent.ThriftMasterRPC: Connected
to master at hadoop48:35872
2013-01-23 17:11:38,687 [main] INFO agent.LogicalNodeManager: creating new
logical node Hadoop47
2013-01-23 17:11:38,819 [main] WARN agent.FlumeNode: Flume is using Hadoop
core 1.0.4 which does not support Security / Authentication: null
2013-01-23 17:11:43,529 [Heartbeat] INFO agent.LogicalNodeManager: creating
new logical node agent2
2013-01-23 17:11:43,536 [logicalNode agent2-18] ERROR
connector.DirectDriver: Closing down due to exception on open calls
2013-01-23 17:11:43,537 [logicalNode agent2-18] INFO
connector.DirectDriver: Connector logicalNode agent2-18 exited with error:
Attempting to open a stub sink 'fail'!
2013-01-23 17:11:43,537 [logicalNode agent2-18] ERROR
connector.DirectDriver: Exiting driver logicalNode agent2-18 in error state
TailSource | fail because Attempting to open a stub sink 'fail'!
2013-01-23 17:11:43,540 [TailThread-0] INFO text.TailSource: TailThread has
exited
2013-01-23 17:11:43,553 [Check config] INFO agent.LogicalNode: Node config
successfully set to com.cloudera.flume.conf.FlumeConfigData@76a0f9cb
2013-01-23 17:11:43,554 [logicalNode agent2-20] ERROR
connector.DirectDriver: Closing down due to exception on open calls
2013-01-23 17:11:43,554 [logicalNode agent2-20] INFO
connector.DirectDriver: Connector logicalNode agent2-20 exited with error:
Attempting to open a stub sink 'fail'!
2013-01-23 17:11:43,554 [logicalNode agent2-20] ERROR
connector.DirectDriver: Exiting driver logicalNode agent2-20 in error state
TailSource | fail because Attempting to open a stub sink 'fail'!
2013-01-23 17:11:43,554 [TailThread-1] INFO text.TailSource: TailThread has
exited

http://hadoop48:35871/flumemaster.jsp
Node status logical node physical nodehost name statusversionlast seen
delta (s) last seenHadoop46 Hadoop46Hadoop46IDLEnone 2Wed Jan 23 17:07:05
CST 2013 Hadoop47Hadoop47Hadoop47 IDLEnone1Wed Jan 23 17:07:06 CST 2013
Hadoop48Hadoop48Hadoop48 IDLEWed Jan 23 16:49:27 CST 2013 0Wed Jan 23
17:07:08 CST 2013WX-wuwb.TKOffice.local WX-wuwb.TKOffice.local
WX-wuwb.TKOffice.localERROR Wed Jan 23 16:49:27 CST 20130Wed Jan 23
17:07:07 CST 2013 agent2Hadoop47Hadoop47ERROR Wed Jan 23 16:49:27 CST
20131Wed Jan 23 17:07:06 CST 2013
aggent1Hadoop46 Hadoop46IDLEnone2 Wed Jan 23 17:07:05 CST 2013co1co1Hadoop48
LOSTWed Jan 23 09:56:
+
Jeong-shik Jang 2013-01-23, 14:36
+
周梦想 2013-01-25, 07:32