Lichen
2012-09-29, 07:57
Brock Noland
2012-09-29, 14:18
Mike Percy
2012-09-30, 02:49
Hari Shreedharan
2012-09-30, 03:38
Lichen
2012-10-08, 07:37
Mike Percy
2012-10-09, 02:58
Lichen
2012-10-09, 03:48
Mike Percy
2012-10-12, 00:47
Yanzhi.liu
2012-09-29, 08:13
-
Unable to use HDFS sinkLichen 2012-09-29, 07:57
Hi. I'm using flume 1.2 with hadoop-0.20.2 but unable to use HDFS sink. Flume stuck during creating HDFS sink, which is OK using file-roll or logger sinks. I've included $HADOOP_HOME and $HADOOP_HOME/lib in FLUME_CLASSPATH. Is it because my hadoop version not compatible?
Configuration file:
agent_foo.sources = netcat-source
agent_foo.sinks = hdfs-sink
agent_foo.channels = mem-channel-1
# properties of netcat-source
agent_foo.sources.netcat-source.channels = mem-channel-1
agent_foo.sources.netcat-source.type = netcat
agent_foo.sources.netcat-source.bind = localhost
agent_foo.sources.netcat-source.port = 44444
# properties of hdfs-sink
agent_foo.sinks.hdfs-sink.channel = mem-channel-1
agent_foo.sinks.hdfs-sink.type = hdfs
agent_foo.sinks.hdfs-sink.hdfs.path = hdfs://hadoop0:9000/flume/
# properties of mem-channel-1
agent_foo.channels.mem-channel-1.type = memory
agent_foo.channels.mem-channel-1.capacity = 1000
agent_foo.channels.mem-channel-1.transactionCapacity = 100
Log:
$ bin/flume-ng agent -c conf/ -f conf/test.conf -n agent_foo
2012-09-29 15:19:47,765 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1
2012-09-29 15:19:47,768 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent_foo
2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] Node manager starting
2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 11
2012-09-29 15:19:47,774 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting
2012-09-29 15:19:47,776 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)] Reloading configuration file:conf/test.conf
2012-09-29 15:19:47,780 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] Added sinks: hdfs-sink Agent: agent_foo
2012-09-29 15:19:47,789 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)] Post-validation flume configuration contains configuration for agents: [agent_foo]
2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)] Creating channels
2012-09-29 15:19:47,858 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)] Monitoried counter group for type: CHANNEL, name: mem-channel-1, registered successfully.
2012-09-29 15:19:47,859 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)] created channel mem-channel-1
2012-09-29 15:19:47,879 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)] Creating instance of sink: hdfs-sink, type: hdfs
There is no more output after the last line.
Configuration file:
agent_foo.sources = netcat-source
agent_foo.sinks = hdfs-sink
agent_foo.channels = mem-channel-1
# properties of netcat-source
agent_foo.sources.netcat-source.channels = mem-channel-1
agent_foo.sources.netcat-source.type = netcat
agent_foo.sources.netcat-source.bind = localhost
agent_foo.sources.netcat-source.port = 44444
# properties of hdfs-sink
agent_foo.sinks.hdfs-sink.channel = mem-channel-1
agent_foo.sinks.hdfs-sink.type = hdfs
agent_foo.sinks.hdfs-sink.hdfs.path = hdfs://hadoop0:9000/flume/
# properties of mem-channel-1
agent_foo.channels.mem-channel-1.type = memory
agent_foo.channels.mem-channel-1.capacity = 1000
agent_foo.channels.mem-channel-1.transactionCapacity = 100
Log:
$ bin/flume-ng agent -c conf/ -f conf/test.conf -n agent_foo
2012-09-29 15:19:47,765 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1
2012-09-29 15:19:47,768 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent_foo
2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] Node manager starting
2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 11
2012-09-29 15:19:47,774 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting
2012-09-29 15:19:47,776 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)] Reloading configuration file:conf/test.conf
2012-09-29 15:19:47,780 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] Added sinks: hdfs-sink Agent: agent_foo
2012-09-29 15:19:47,789 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)] Post-validation flume configuration contains configuration for agents: [agent_foo]
2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)] Creating channels
2012-09-29 15:19:47,858 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)] Monitoried counter group for type: CHANNEL, name: mem-channel-1, registered successfully.
2012-09-29 15:19:47,859 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)] created channel mem-channel-1
2012-09-29 15:19:47,879 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)] Creating instance of sink: hdfs-sink, type: hdfs
There is no more output after the last line.
+
Lichen 2012-09-29, 07:57
-
Re: Unable to use HDFS sinkBrock Noland 2012-09-29, 14:18
0.20.2 might work but it's very old. I would update to 1.x and try again. If that doesn't work, let us know.
--
Brock Noland
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Saturday, September 29, 2012 at 2:57 AM, Lichen wrote:
>
> Hi. I’m using flume 1.2 with hadoop-0.20.2 but unable to use HDFS sink. Flume stuck during creating HDFS sink, which is OK using file-roll or logger sinks. I’ve included $HADOOP_HOME and $HADOOP_HOME/lib in FLUME_CLASSPATH. Is it because my hadoop version not compatible?
>
>
>
>
>
> Configuration file:
>
>
> agent_foo.sources = netcat-source
>
>
> agent_foo.sinks = hdfs-sink
>
>
> agent_foo.channels = mem-channel-1
>
>
>
>
>
> # properties of netcat-source
>
>
> agent_foo.sources.netcat-source.channels = mem-channel-1
>
>
> agent_foo.sources.netcat-source.type = netcat
>
>
> agent_foo.sources.netcat-source.bind = localhost
>
>
> agent_foo.sources.netcat-source.port = 44444
>
>
>
>
>
> # properties of hdfs-sink
>
>
> agent_foo.sinks.hdfs-sink.channel = mem-channel-1
>
>
> agent_foo.sinks.hdfs-sink.type = hdfs
>
>
> agent_foo.sinks.hdfs-sink.hdfs.path = hdfs://hadoop0:9000/flume/
>
>
>
>
>
> # properties of mem-channel-1
>
>
> agent_foo.channels.mem-channel-1.type = memory
>
>
> agent_foo.channels.mem-channel-1.capacity = 1000
>
>
> agent_foo.channels.mem-channel-1.transactionCapacity = 100
>
>
>
>
>
> Log:
>
>
> $ bin/flume-ng agent -c conf/ -f conf/test.conf -n agent_foo
>
>
> 2012-09-29 15:19:47,765 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1
>
>
> 2012-09-29 15:19:47,768 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent_foo
>
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] Node manager starting
>
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 11
>
>
> 2012-09-29 15:19:47,774 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting
>
>
> 2012-09-29 15:19:47,776 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)] Reloading configuration file:conf/test.conf
>
>
> 2012-09-29 15:19:47,780 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
>
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
>
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] Added sinks: hdfs-sink Agent: agent_foo
>
>
> 2012-09-29 15:19:47,789 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
>
>
> 2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)] Post-validation flume configuration contains configuration for agents: [agent_foo]
>
>
> 2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)] Creating channels
>
>
> 2012-09-29 15:19:47,858 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)] Monitoried counter group for type: CHANNEL, name: mem-channel-1, registered successfully.
--
Brock Noland
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Saturday, September 29, 2012 at 2:57 AM, Lichen wrote:
>
> Hi. I’m using flume 1.2 with hadoop-0.20.2 but unable to use HDFS sink. Flume stuck during creating HDFS sink, which is OK using file-roll or logger sinks. I’ve included $HADOOP_HOME and $HADOOP_HOME/lib in FLUME_CLASSPATH. Is it because my hadoop version not compatible?
>
>
>
>
>
> Configuration file:
>
>
> agent_foo.sources = netcat-source
>
>
> agent_foo.sinks = hdfs-sink
>
>
> agent_foo.channels = mem-channel-1
>
>
>
>
>
> # properties of netcat-source
>
>
> agent_foo.sources.netcat-source.channels = mem-channel-1
>
>
> agent_foo.sources.netcat-source.type = netcat
>
>
> agent_foo.sources.netcat-source.bind = localhost
>
>
> agent_foo.sources.netcat-source.port = 44444
>
>
>
>
>
> # properties of hdfs-sink
>
>
> agent_foo.sinks.hdfs-sink.channel = mem-channel-1
>
>
> agent_foo.sinks.hdfs-sink.type = hdfs
>
>
> agent_foo.sinks.hdfs-sink.hdfs.path = hdfs://hadoop0:9000/flume/
>
>
>
>
>
> # properties of mem-channel-1
>
>
> agent_foo.channels.mem-channel-1.type = memory
>
>
> agent_foo.channels.mem-channel-1.capacity = 1000
>
>
> agent_foo.channels.mem-channel-1.transactionCapacity = 100
>
>
>
>
>
> Log:
>
>
> $ bin/flume-ng agent -c conf/ -f conf/test.conf -n agent_foo
>
>
> 2012-09-29 15:19:47,765 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1
>
>
> 2012-09-29 15:19:47,768 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent_foo
>
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] Node manager starting
>
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 11
>
>
> 2012-09-29 15:19:47,774 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting
>
>
> 2012-09-29 15:19:47,776 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)] Reloading configuration file:conf/test.conf
>
>
> 2012-09-29 15:19:47,780 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
>
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
>
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] Added sinks: hdfs-sink Agent: agent_foo
>
>
> 2012-09-29 15:19:47,789 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
>
>
> 2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)] Post-validation flume configuration contains configuration for agents: [agent_foo]
>
>
> 2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)] Creating channels
>
>
> 2012-09-29 15:19:47,858 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)] Monitoried counter group for type: CHANNEL, name: mem-channel-1, registered successfully.
+
Brock Noland 2012-09-29, 14:18
-
Re: Unable to use HDFS sinkMike Percy 2012-09-30, 02:49
Hi,
Don't use vanilla hadoop 0.20.2 ... you will not get durability.
You must use hadoop 0.20.205, hadoop 1.0.x, hadoop 2.0.x, or CDH3/4 with
Flume NG.
Regards,
Mike
On Sat, Sep 29, 2012 at 12:57 AM, Lichen <[EMAIL PROTECTED]> wrote:
> Hi. I’m using flume 1.2 with hadoop-0.20.2 but unable to use HDFS sink.
> Flume stuck during creating HDFS sink, which is OK using file-roll or
> logger sinks. I’ve included $HADOOP_HOME and $HADOOP_HOME/lib in
> FLUME_CLASSPATH. Is it because my hadoop version not compatible?****
>
> ** **
>
> Configuration file:****
>
> agent_foo.sources = netcat-source****
>
> agent_foo.sinks = hdfs-sink****
>
> agent_foo.channels = mem-channel-1****
>
> ** **
>
> # properties of netcat-source****
>
> agent_foo.sources.netcat-source.channels = mem-channel-1****
>
> agent_foo.sources.netcat-source.type = netcat****
>
> agent_foo.sources.netcat-source.bind = localhost****
>
> agent_foo.sources.netcat-source.port = 44444****
>
> ** **
>
> # properties of hdfs-sink****
>
> agent_foo.sinks.hdfs-sink.channel = mem-channel-1****
>
> agent_foo.sinks.hdfs-sink.type = hdfs****
>
> agent_foo.sinks.hdfs-sink.hdfs.path = hdfs://hadoop0:9000/flume/****
>
> ** **
>
> # properties of mem-channel-1****
>
> agent_foo.channels.mem-channel-1.type = memory****
>
> agent_foo.channels.mem-channel-1.capacity = 1000****
>
> agent_foo.channels.mem-channel-1.transactionCapacity = 100****
>
> ** **
>
> Log:****
>
> $ bin/flume-ng agent -c conf/ -f conf/test.conf -n agent_foo****
>
> 2012-09-29 15:19:47,765 (main) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 1****
>
> 2012-09-29 15:19:47,768 (main) [INFO -
> org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node
> starting - agent_foo****
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)]
> Node manager starting****
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 11****
>
> 2012-09-29 15:19:47,774 (lifecycleSupervisor-1-2) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
> Configuration provider starting****
>
> 2012-09-29 15:19:47,776 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)]
> Reloading configuration file:conf/test.conf****
>
> 2012-09-29 15:19:47,780 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
> Processing:hdfs-sink****
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
> Processing:hdfs-sink****
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)]
> Added sinks: hdfs-sink Agent: agent_foo****
>
> 2012-09-29 15:19:47,789 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
> Processing:hdfs-sink****
>
> 2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)]
> Post-validation flume configuration contains configuration for agents:
> [agent_foo]****
>
> 2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)]
> Creating channels****
>
> 2012-09-29 15:19:47,858 (conf-file-poller-0) [INFO -
> org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)]
Don't use vanilla hadoop 0.20.2 ... you will not get durability.
You must use hadoop 0.20.205, hadoop 1.0.x, hadoop 2.0.x, or CDH3/4 with
Flume NG.
Regards,
Mike
On Sat, Sep 29, 2012 at 12:57 AM, Lichen <[EMAIL PROTECTED]> wrote:
> Hi. I’m using flume 1.2 with hadoop-0.20.2 but unable to use HDFS sink.
> Flume stuck during creating HDFS sink, which is OK using file-roll or
> logger sinks. I’ve included $HADOOP_HOME and $HADOOP_HOME/lib in
> FLUME_CLASSPATH. Is it because my hadoop version not compatible?****
>
> ** **
>
> Configuration file:****
>
> agent_foo.sources = netcat-source****
>
> agent_foo.sinks = hdfs-sink****
>
> agent_foo.channels = mem-channel-1****
>
> ** **
>
> # properties of netcat-source****
>
> agent_foo.sources.netcat-source.channels = mem-channel-1****
>
> agent_foo.sources.netcat-source.type = netcat****
>
> agent_foo.sources.netcat-source.bind = localhost****
>
> agent_foo.sources.netcat-source.port = 44444****
>
> ** **
>
> # properties of hdfs-sink****
>
> agent_foo.sinks.hdfs-sink.channel = mem-channel-1****
>
> agent_foo.sinks.hdfs-sink.type = hdfs****
>
> agent_foo.sinks.hdfs-sink.hdfs.path = hdfs://hadoop0:9000/flume/****
>
> ** **
>
> # properties of mem-channel-1****
>
> agent_foo.channels.mem-channel-1.type = memory****
>
> agent_foo.channels.mem-channel-1.capacity = 1000****
>
> agent_foo.channels.mem-channel-1.transactionCapacity = 100****
>
> ** **
>
> Log:****
>
> $ bin/flume-ng agent -c conf/ -f conf/test.conf -n agent_foo****
>
> 2012-09-29 15:19:47,765 (main) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 1****
>
> 2012-09-29 15:19:47,768 (main) [INFO -
> org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node
> starting - agent_foo****
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)]
> Node manager starting****
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 11****
>
> 2012-09-29 15:19:47,774 (lifecycleSupervisor-1-2) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
> Configuration provider starting****
>
> 2012-09-29 15:19:47,776 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)]
> Reloading configuration file:conf/test.conf****
>
> 2012-09-29 15:19:47,780 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
> Processing:hdfs-sink****
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
> Processing:hdfs-sink****
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)]
> Added sinks: hdfs-sink Agent: agent_foo****
>
> 2012-09-29 15:19:47,789 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
> Processing:hdfs-sink****
>
> 2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)]
> Post-validation flume configuration contains configuration for agents:
> [agent_foo]****
>
> 2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)]
> Creating channels****
>
> 2012-09-29 15:19:47,858 (conf-file-poller-0) [INFO -
> org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)]
+
Mike Percy 2012-09-30, 02:49
-
Re: Unable to use HDFS sinkHari Shreedharan 2012-09-30, 03:38
Note that Flume will not work directly against Hadoop-2.0.x yet, since
flume depends on hbase which does not have a Hadoop-2.0.x compatible
release yet.
On Saturday, September 29, 2012, Mike Percy wrote:
> Hi,
> Don't use vanilla hadoop 0.20.2 ... you will not get durability.
>
> You must use hadoop 0.20.205, hadoop 1.0.x, hadoop 2.0.x, or CDH3/4 with
> Flume NG.
>
> Regards,
> Mike
>
> On Sat, Sep 29, 2012 at 12:57 AM, Lichen <[EMAIL PROTECTED]> wrote:
>
> Hi. I’m using flume 1.2 with hadoop-0.20.2 but unable to use HDFS sink.
> Flume stuck during creating HDFS sink, which is OK using file-roll or
> logger sinks. I’ve included $HADOOP_HOME and $HADOOP_HOME/lib in
> FLUME_CLASSPATH. Is it because my hadoop version not compatible?****
>
> ** **
>
> Configuration file:****
>
> agent_foo.sources = netcat-source****
>
> agent_foo.sinks = hdfs-sink****
>
> agent_foo.channels = mem-channel-1****
>
> ** **
>
> # properties of netcat-source****
>
> agent_foo.sources.netcat-source.channels = mem-channel-1****
>
> agent_foo.sources.netcat-source.type = netcat****
>
> agent_foo.sources.netcat-source.bind = localhost****
>
> agent_foo.sources.netcat-source.port = 44444****
>
> ** **
>
> # properties of hdfs-sink****
>
> agent_foo.sinks.hdfs-sink.channel = mem-channel-1****
>
> agent_foo.sinks.hdfs-sink.type = hdfs****
>
> agent_foo.sinks.hdfs-sink.hdfs.path = hdfs://hadoop0:9000/flume/****
>
> ** **
>
> # properties of mem-channel-1****
>
> agent_foo.channels.mem-channel-1.type = memory****
>
> agent_foo.channels.mem-channel-1.capacity = 1000****
>
> agent_foo.channels.mem-channel-1.transactionCapacity = 100****
>
> ** **
>
> Log:****
>
> $ bin/flume-ng agent -c conf/ -f conf/test.conf -n agent_foo****
>
> 2012-09-29 15:19:47,765 (main) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 1****
>
> 2012-09-29 15:19:47,768 (main) [INFO -
> org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node
> starting - agent_foo****
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)]
> Node manager starting****
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 11****
>
> 2012-09-29 15:19:47,774 (lifecycleSupervisor-1-2) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
> Configuration provider starting****
>
> 2012-09-29 15:19:47,776 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)]
> Reloading configuration file:conf/test.conf****
>
> 2012-09-29 15:19:47,780 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
> Processing:hdfs-sink****
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addPrope
>
>
flume depends on hbase which does not have a Hadoop-2.0.x compatible
release yet.
On Saturday, September 29, 2012, Mike Percy wrote:
> Hi,
> Don't use vanilla hadoop 0.20.2 ... you will not get durability.
>
> You must use hadoop 0.20.205, hadoop 1.0.x, hadoop 2.0.x, or CDH3/4 with
> Flume NG.
>
> Regards,
> Mike
>
> On Sat, Sep 29, 2012 at 12:57 AM, Lichen <[EMAIL PROTECTED]> wrote:
>
> Hi. I’m using flume 1.2 with hadoop-0.20.2 but unable to use HDFS sink.
> Flume stuck during creating HDFS sink, which is OK using file-roll or
> logger sinks. I’ve included $HADOOP_HOME and $HADOOP_HOME/lib in
> FLUME_CLASSPATH. Is it because my hadoop version not compatible?****
>
> ** **
>
> Configuration file:****
>
> agent_foo.sources = netcat-source****
>
> agent_foo.sinks = hdfs-sink****
>
> agent_foo.channels = mem-channel-1****
>
> ** **
>
> # properties of netcat-source****
>
> agent_foo.sources.netcat-source.channels = mem-channel-1****
>
> agent_foo.sources.netcat-source.type = netcat****
>
> agent_foo.sources.netcat-source.bind = localhost****
>
> agent_foo.sources.netcat-source.port = 44444****
>
> ** **
>
> # properties of hdfs-sink****
>
> agent_foo.sinks.hdfs-sink.channel = mem-channel-1****
>
> agent_foo.sinks.hdfs-sink.type = hdfs****
>
> agent_foo.sinks.hdfs-sink.hdfs.path = hdfs://hadoop0:9000/flume/****
>
> ** **
>
> # properties of mem-channel-1****
>
> agent_foo.channels.mem-channel-1.type = memory****
>
> agent_foo.channels.mem-channel-1.capacity = 1000****
>
> agent_foo.channels.mem-channel-1.transactionCapacity = 100****
>
> ** **
>
> Log:****
>
> $ bin/flume-ng agent -c conf/ -f conf/test.conf -n agent_foo****
>
> 2012-09-29 15:19:47,765 (main) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 1****
>
> 2012-09-29 15:19:47,768 (main) [INFO -
> org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node
> starting - agent_foo****
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)]
> Node manager starting****
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 11****
>
> 2012-09-29 15:19:47,774 (lifecycleSupervisor-1-2) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
> Configuration provider starting****
>
> 2012-09-29 15:19:47,776 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)]
> Reloading configuration file:conf/test.conf****
>
> 2012-09-29 15:19:47,780 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
> Processing:hdfs-sink****
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addPrope
>
>
+
Hari Shreedharan 2012-09-30, 03:38
-
RE: Unable to use HDFS sinkLichen 2012-10-08, 07:37
Hi Mike,
I’ve tried hadoop-1.0.3 which doesn’t work as before. Nothing happens after
[INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)] Creating instance of sink: hdfs-sink, type: hdfs
I can execute $HADOOP_HOME/bin/hadoop classpath. Don’t know why the HDFS sink is not working.
Chen
On Sat, Sep 30, 2012 at 10:49 AM, Mike Percy <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote::
Hi,
Don't use vanilla hadoop 0.20.2 ... you will not get durability.
You must use hadoop 0.20.205, hadoop 1.0.x, hadoop 2.0.x, or CDH3/4 with Flume NG.
Regards,
Mike
On Sat, Sep 29, 2012 at 12:57 AM, Lichen <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi. I’m using flume 1.2 with hadoop-0.20.2 but unable to use HDFS sink. Flume stuck during creating HDFS sink, which is OK using file-roll or logger sinks. I’ve included $HADOOP_HOME and $HADOOP_HOME/lib in FLUME_CLASSPATH. Is it because my hadoop version not compatible?
Configuration file:
agent_foo.sources = netcat-source
agent_foo.sinks = hdfs-sink
agent_foo.channels = mem-channel-1
# properties of netcat-source
agent_foo.sources.netcat-source.channels = mem-channel-1
agent_foo.sources.netcat-source.type = netcat
agent_foo.sources.netcat-source.bind = localhost
agent_foo.sources.netcat-source.port = 44444
# properties of hdfs-sink
agent_foo.sinks.hdfs-sink.channel = mem-channel-1
agent_foo.sinks.hdfs-sink.type = hdfs
agent_foo.sinks.hdfs-sink.hdfs.path = hdfs://hadoop0:9000/flume/
# properties of mem-channel-1
agent_foo.channels.mem-channel-1.type = memory
agent_foo.channels.mem-channel-1.capacity = 1000
agent_foo.channels.mem-channel-1.transactionCapacity = 100
Log:
$ bin/flume-ng agent -c conf/ -f conf/test.conf -n agent_foo
2012-09-29 15:19:47,765 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1
2012-09-29 15:19:47,768 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent_foo
2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] Node manager starting
2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 11
2012-09-29 15:19:47,774 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting
2012-09-29 15:19:47,776 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)] Reloading configuration file:conf/test.conf
2012-09-29 15:19:47,780 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] Added sinks: hdfs-sink Agent: agent_foo
2012-09-29 15:19:47,789 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)] Post-validation flume configuration contains configuration for agents: [agent_foo]
2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)] Creating channels
2012-09-29 15:19:47,858 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)] Monitoried counter group for type: CHANNEL, name: mem-channel-1, registered successfully.
2012-09-29 15:19:47,859 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)] created channel mem-channel-1
2012-09-29 15:19:47,879 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)] Creating instance of sink: hdfs-sink, type: hdfs
There is no more output after the last line.
I’ve tried hadoop-1.0.3 which doesn’t work as before. Nothing happens after
[INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)] Creating instance of sink: hdfs-sink, type: hdfs
I can execute $HADOOP_HOME/bin/hadoop classpath. Don’t know why the HDFS sink is not working.
Chen
On Sat, Sep 30, 2012 at 10:49 AM, Mike Percy <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote::
Hi,
Don't use vanilla hadoop 0.20.2 ... you will not get durability.
You must use hadoop 0.20.205, hadoop 1.0.x, hadoop 2.0.x, or CDH3/4 with Flume NG.
Regards,
Mike
On Sat, Sep 29, 2012 at 12:57 AM, Lichen <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi. I’m using flume 1.2 with hadoop-0.20.2 but unable to use HDFS sink. Flume stuck during creating HDFS sink, which is OK using file-roll or logger sinks. I’ve included $HADOOP_HOME and $HADOOP_HOME/lib in FLUME_CLASSPATH. Is it because my hadoop version not compatible?
Configuration file:
agent_foo.sources = netcat-source
agent_foo.sinks = hdfs-sink
agent_foo.channels = mem-channel-1
# properties of netcat-source
agent_foo.sources.netcat-source.channels = mem-channel-1
agent_foo.sources.netcat-source.type = netcat
agent_foo.sources.netcat-source.bind = localhost
agent_foo.sources.netcat-source.port = 44444
# properties of hdfs-sink
agent_foo.sinks.hdfs-sink.channel = mem-channel-1
agent_foo.sinks.hdfs-sink.type = hdfs
agent_foo.sinks.hdfs-sink.hdfs.path = hdfs://hadoop0:9000/flume/
# properties of mem-channel-1
agent_foo.channels.mem-channel-1.type = memory
agent_foo.channels.mem-channel-1.capacity = 1000
agent_foo.channels.mem-channel-1.transactionCapacity = 100
Log:
$ bin/flume-ng agent -c conf/ -f conf/test.conf -n agent_foo
2012-09-29 15:19:47,765 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1
2012-09-29 15:19:47,768 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent_foo
2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] Node manager starting
2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 11
2012-09-29 15:19:47,774 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting
2012-09-29 15:19:47,776 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)] Reloading configuration file:conf/test.conf
2012-09-29 15:19:47,780 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] Added sinks: hdfs-sink Agent: agent_foo
2012-09-29 15:19:47,789 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)] Post-validation flume configuration contains configuration for agents: [agent_foo]
2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)] Creating channels
2012-09-29 15:19:47,858 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)] Monitoried counter group for type: CHANNEL, name: mem-channel-1, registered successfully.
2012-09-29 15:19:47,859 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)] created channel mem-channel-1
2012-09-29 15:19:47,879 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)] Creating instance of sink: hdfs-sink, type: hdfs
There is no more output after the last line.
+
Lichen 2012-10-08, 07:37
-
Re: Unable to use HDFS sinkMike Percy 2012-10-09, 02:58
Chen if you can send jstack output that could help. Or do: sudo kill -3
<pid> on Flume and send the output from the Flume stdout, which is the same
thing.
Regards
Mike
On Mon, Oct 8, 2012 at 12:37 AM, Lichen <[EMAIL PROTECTED]> wrote:
> Hi Mike,****
>
> ** **
>
> I’ve tried hadoop-1.0.3 which doesn’t work as before. Nothing happens after
> ****
>
> [INFO -
> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)]
> Creating instance of sink: hdfs-sink, type: hdfs****
>
> I can execute $HADOOP_HOME/bin/hadoop classpath. Don’t know why the HDFS
> sink is not working.****
>
> ** **
>
> Chen****
>
> ** **
>
> On Sat, Sep 30, 2012 at 10:49 AM, Mike Percy <[EMAIL PROTECTED]> wrote::**
> **
>
> ** **
>
> Hi,****
>
> Don't use vanilla hadoop 0.20.2 ... you will not get durability.****
>
> ** **
>
> You must use hadoop 0.20.205, hadoop 1.0.x, hadoop 2.0.x, or CDH3/4 with
> Flume NG.****
>
> ** **
>
> Regards,****
>
> Mike****
>
> ** **
>
> On Sat, Sep 29, 2012 at 12:57 AM, Lichen <[EMAIL PROTECTED]> wrote:***
> *
>
> Hi. I’m using flume 1.2 with hadoop-0.20.2 but unable to use HDFS sink.
> Flume stuck during creating HDFS sink, which is OK using file-roll or
> logger sinks. I’ve included $HADOOP_HOME and $HADOOP_HOME/lib in
> FLUME_CLASSPATH. Is it because my hadoop version not compatible?****
>
> ****
>
> Configuration file:****
>
> agent_foo.sources = netcat-source****
>
> agent_foo.sinks = hdfs-sink****
>
> agent_foo.channels = mem-channel-1****
>
> ****
>
> # properties of netcat-source****
>
> agent_foo.sources.netcat-source.channels = mem-channel-1****
>
> agent_foo.sources.netcat-source.type = netcat****
>
> agent_foo.sources.netcat-source.bind = localhost****
>
> agent_foo.sources.netcat-source.port = 44444****
>
> ****
>
> # properties of hdfs-sink****
>
> agent_foo.sinks.hdfs-sink.channel = mem-channel-1****
>
> agent_foo.sinks.hdfs-sink.type = hdfs****
>
> agent_foo.sinks.hdfs-sink.hdfs.path = hdfs://hadoop0:9000/flume/****
>
> ****
>
> # properties of mem-channel-1****
>
> agent_foo.channels.mem-channel-1.type = memory****
>
> agent_foo.channels.mem-channel-1.capacity = 1000****
>
> agent_foo.channels.mem-channel-1.transactionCapacity = 100****
>
> ****
>
> Log:****
>
> $ bin/flume-ng agent -c conf/ -f conf/test.conf -n agent_foo****
>
> 2012-09-29 15:19:47,765 (main) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 1****
>
> 2012-09-29 15:19:47,768 (main) [INFO -
> org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node
> starting - agent_foo****
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)]
> Node manager starting****
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 11****
>
> 2012-09-29 15:19:47,774 (lifecycleSupervisor-1-2) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
> Configuration provider starting****
>
> 2012-09-29 15:19:47,776 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)]
> Reloading configuration file:conf/test.conf****
>
> 2012-09-29 15:19:47,780 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
> Processing:hdfs-sink****
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
> Processing:hdfs-sink****
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)]
> Added sinks: hdfs-sink Agent: agent_foo****
<pid> on Flume and send the output from the Flume stdout, which is the same
thing.
Regards
Mike
On Mon, Oct 8, 2012 at 12:37 AM, Lichen <[EMAIL PROTECTED]> wrote:
> Hi Mike,****
>
> ** **
>
> I’ve tried hadoop-1.0.3 which doesn’t work as before. Nothing happens after
> ****
>
> [INFO -
> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)]
> Creating instance of sink: hdfs-sink, type: hdfs****
>
> I can execute $HADOOP_HOME/bin/hadoop classpath. Don’t know why the HDFS
> sink is not working.****
>
> ** **
>
> Chen****
>
> ** **
>
> On Sat, Sep 30, 2012 at 10:49 AM, Mike Percy <[EMAIL PROTECTED]> wrote::**
> **
>
> ** **
>
> Hi,****
>
> Don't use vanilla hadoop 0.20.2 ... you will not get durability.****
>
> ** **
>
> You must use hadoop 0.20.205, hadoop 1.0.x, hadoop 2.0.x, or CDH3/4 with
> Flume NG.****
>
> ** **
>
> Regards,****
>
> Mike****
>
> ** **
>
> On Sat, Sep 29, 2012 at 12:57 AM, Lichen <[EMAIL PROTECTED]> wrote:***
> *
>
> Hi. I’m using flume 1.2 with hadoop-0.20.2 but unable to use HDFS sink.
> Flume stuck during creating HDFS sink, which is OK using file-roll or
> logger sinks. I’ve included $HADOOP_HOME and $HADOOP_HOME/lib in
> FLUME_CLASSPATH. Is it because my hadoop version not compatible?****
>
> ****
>
> Configuration file:****
>
> agent_foo.sources = netcat-source****
>
> agent_foo.sinks = hdfs-sink****
>
> agent_foo.channels = mem-channel-1****
>
> ****
>
> # properties of netcat-source****
>
> agent_foo.sources.netcat-source.channels = mem-channel-1****
>
> agent_foo.sources.netcat-source.type = netcat****
>
> agent_foo.sources.netcat-source.bind = localhost****
>
> agent_foo.sources.netcat-source.port = 44444****
>
> ****
>
> # properties of hdfs-sink****
>
> agent_foo.sinks.hdfs-sink.channel = mem-channel-1****
>
> agent_foo.sinks.hdfs-sink.type = hdfs****
>
> agent_foo.sinks.hdfs-sink.hdfs.path = hdfs://hadoop0:9000/flume/****
>
> ****
>
> # properties of mem-channel-1****
>
> agent_foo.channels.mem-channel-1.type = memory****
>
> agent_foo.channels.mem-channel-1.capacity = 1000****
>
> agent_foo.channels.mem-channel-1.transactionCapacity = 100****
>
> ****
>
> Log:****
>
> $ bin/flume-ng agent -c conf/ -f conf/test.conf -n agent_foo****
>
> 2012-09-29 15:19:47,765 (main) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 1****
>
> 2012-09-29 15:19:47,768 (main) [INFO -
> org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node
> starting - agent_foo****
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)]
> Node manager starting****
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 11****
>
> 2012-09-29 15:19:47,774 (lifecycleSupervisor-1-2) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
> Configuration provider starting****
>
> 2012-09-29 15:19:47,776 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)]
> Reloading configuration file:conf/test.conf****
>
> 2012-09-29 15:19:47,780 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
> Processing:hdfs-sink****
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
> Processing:hdfs-sink****
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)]
> Added sinks: hdfs-sink Agent: agent_foo****
+
Mike Percy 2012-10-09, 02:58
-
RE: Unable to use HDFS sinkLichen 2012-10-09, 03:48
Mike, I’ve dumped flume and get this:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode):
"Attach Listener" daemon prio=10 tid=0x00002aaab5088000 nid=0x3229 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"conf-file-poller-0" prio=10 tid=0x00002aaab50d6000 nid=0x3210 waiting on condition [0x0000000041139000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab49f8aa8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"lifecycleSupervisor-1-0" prio=10 tid=0x00002aaab50fe000 nid=0x320f waiting on condition [0x0000000041038000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab49ff488> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"lifecycleSupervisor-1-2" prio=10 tid=0x00002aaab5cf7800 nid=0x320e waiting on condition [0x0000000040f37000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab4a23be8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"lifecycleSupervisor-1-1" prio=10 tid=0x00002aaab5cf7000 nid=0x320d waiting on condition [0x0000000040e36000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab4a23be8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"lifecycleSupervisor-1-0" prio=10 tid=0x00002aaab5cf6000 nid=0x320c waiting on condition [0x0000000040d35000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab4a23be8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"Low Memory Detector" daemon prio=10 tid=0x0000
Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode):
"Attach Listener" daemon prio=10 tid=0x00002aaab5088000 nid=0x3229 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"conf-file-poller-0" prio=10 tid=0x00002aaab50d6000 nid=0x3210 waiting on condition [0x0000000041139000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab49f8aa8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"lifecycleSupervisor-1-0" prio=10 tid=0x00002aaab50fe000 nid=0x320f waiting on condition [0x0000000041038000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab49ff488> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"lifecycleSupervisor-1-2" prio=10 tid=0x00002aaab5cf7800 nid=0x320e waiting on condition [0x0000000040f37000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab4a23be8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"lifecycleSupervisor-1-1" prio=10 tid=0x00002aaab5cf7000 nid=0x320d waiting on condition [0x0000000040e36000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab4a23be8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"lifecycleSupervisor-1-0" prio=10 tid=0x00002aaab5cf6000 nid=0x320c waiting on condition [0x0000000040d35000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab4a23be8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"Low Memory Detector" daemon prio=10 tid=0x0000
+
Lichen 2012-10-09, 03:48
-
Re: Unable to use HDFS sinkMike Percy 2012-10-12, 00:47
Hi Chen,
I don't see the SinkRunner thread so it didn't get very far. I'm not sure
what went wrong but it looks like some error was swallowed somewhere by
some Thread maybe. Seems strange. But it isn't immediately obvious to me
what went wrong since if you are using a recent version of Flume it should
print an error to the log if it gets a ClassNotFoundException, say if it
can't find the Hadoop classes.
Regards,
Mike
On Mon, Oct 8, 2012 a
t 8:48 PM, Lichen <[EMAIL PROTECTED]> wrote:
> Mike, I’ve dumped flume and get this:****
>
> ** **
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode):*
> ***
>
> ** **
>
> "Attach Listener" daemon prio=10 tid=0x00002aaab5088000 nid=0x3229 waiting
> on condition [0x0000000000000000]****
>
> java.lang.Thread.State: RUNNABLE****
>
> ** **
>
> Locked ownable synchronizers:****
>
> - None****
>
> ** **
>
> "conf-file-poller-0" prio=10 tid=0x00002aaab50d6000 nid=0x3210 waiting on
> condition [0x0000000041139000]****
>
> java.lang.Thread.State: WAITING (parking)****
>
> at sun.misc.Unsafe.park(Native Method)****
>
> - parking to wait for <0x00002aaab49f8aa8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)****
>
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)****
>
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> ****
>
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)****
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
> ****
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
> ****
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> ****
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> ****
>
> at java.lang.Thread.run(Thread.java:619)****
>
> ** **
>
> Locked ownable synchronizers:****
>
> - None****
>
> ** **
>
> "lifecycleSupervisor-1-0" prio=10 tid=0x00002aaab50fe000 nid=0x320f
> waiting on condition [0x0000000041038000]****
>
> java.lang.Thread.State: TIMED_WAITING (parking)****
>
> at sun.misc.Unsafe.park(Native Method)****
>
> - parking to wait for <0x00002aaab49ff488> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)****
>
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)****
>
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
> ****
>
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)****
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
> ****
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
> ****
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> ****
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> ****
>
> at java.lang.Thread.run(Thread.java:619)****
>
> ** **
>
> Locked ownable synchronizers:****
>
> - None****
>
> ** **
>
> "lifecycleSupervisor-1-2" prio=10 tid=0x00002aaab5cf7800 nid=0x320e
> waiting on condition [0x0000000040f37000]****
>
> java.lang.Thread.State: TIMED_WAITING (parking)****
>
> at sun.misc.Unsafe.park(Native Method)****
>
> - parking to wait for <0x00002aaab4a23be8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)****
>
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)****
>
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
I don't see the SinkRunner thread so it didn't get very far. I'm not sure
what went wrong but it looks like some error was swallowed somewhere by
some Thread maybe. Seems strange. But it isn't immediately obvious to me
what went wrong since if you are using a recent version of Flume it should
print an error to the log if it gets a ClassNotFoundException, say if it
can't find the Hadoop classes.
Regards,
Mike
On Mon, Oct 8, 2012 a
t 8:48 PM, Lichen <[EMAIL PROTECTED]> wrote:
> Mike, I’ve dumped flume and get this:****
>
> ** **
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode):*
> ***
>
> ** **
>
> "Attach Listener" daemon prio=10 tid=0x00002aaab5088000 nid=0x3229 waiting
> on condition [0x0000000000000000]****
>
> java.lang.Thread.State: RUNNABLE****
>
> ** **
>
> Locked ownable synchronizers:****
>
> - None****
>
> ** **
>
> "conf-file-poller-0" prio=10 tid=0x00002aaab50d6000 nid=0x3210 waiting on
> condition [0x0000000041139000]****
>
> java.lang.Thread.State: WAITING (parking)****
>
> at sun.misc.Unsafe.park(Native Method)****
>
> - parking to wait for <0x00002aaab49f8aa8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)****
>
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)****
>
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> ****
>
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)****
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
> ****
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
> ****
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> ****
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> ****
>
> at java.lang.Thread.run(Thread.java:619)****
>
> ** **
>
> Locked ownable synchronizers:****
>
> - None****
>
> ** **
>
> "lifecycleSupervisor-1-0" prio=10 tid=0x00002aaab50fe000 nid=0x320f
> waiting on condition [0x0000000041038000]****
>
> java.lang.Thread.State: TIMED_WAITING (parking)****
>
> at sun.misc.Unsafe.park(Native Method)****
>
> - parking to wait for <0x00002aaab49ff488> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)****
>
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)****
>
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
> ****
>
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)****
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
> ****
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
> ****
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> ****
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> ****
>
> at java.lang.Thread.run(Thread.java:619)****
>
> ** **
>
> Locked ownable synchronizers:****
>
> - None****
>
> ** **
>
> "lifecycleSupervisor-1-2" prio=10 tid=0x00002aaab5cf7800 nid=0x320e
> waiting on condition [0x0000000040f37000]****
>
> java.lang.Thread.State: TIMED_WAITING (parking)****
>
> at sun.misc.Unsafe.park(Native Method)****
>
> - parking to wait for <0x00002aaab4a23be8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)****
>
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)****
>
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
+
Mike Percy 2012-10-12, 00:47
-
Re:Unable to use HDFS sinkYanzhi.liu 2012-09-29, 08:13
Hello:
Upgrade hadoop enough.
------------------ Original ------------------
From: "Lichen"<[EMAIL PROTECTED]>;
Date: Sat, Sep 29, 2012 03:57 PM
To: "[EMAIL PROTECTED]"<[EMAIL PROTECTED]>;
Subject: Unable to use HDFS sink
Hi. I’m using flume 1.2 with hadoop-0.20.2 but unable to use HDFS sink. Flume stuck during creating HDFS sink, which is OK using file-roll or logger sinks. I’ve included $HADOOP_HOME and $HADOOP_HOME/lib in FLUME_CLASSPATH. Is it because my hadoop version not compatible?
Configuration file:
agent_foo.sources = netcat-source
agent_foo.sinks = hdfs-sink
agent_foo.channels = mem-channel-1
# properties of netcat-source
agent_foo.sources.netcat-source.channels = mem-channel-1
agent_foo.sources.netcat-source.type = netcat
agent_foo.sources.netcat-source.bind = localhost
agent_foo.sources.netcat-source.port = 44444
# properties of hdfs-sink
agent_foo.sinks.hdfs-sink.channel = mem-channel-1
agent_foo.sinks.hdfs-sink.type = hdfs
agent_foo.sinks.hdfs-sink.hdfs.path = hdfs://hadoop0:9000/flume/
# properties of mem-channel-1
agent_foo.channels.mem-channel-1.type = memory
agent_foo.channels.mem-channel-1.capacity = 1000
agent_foo.channels.mem-channel-1.transactionCapacity = 100
Log:
$ bin/flume-ng agent -c conf/ -f conf/test.conf -n agent_foo
2012-09-29 15:19:47,765 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1
2012-09-29 15:19:47,768 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent_foo
2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] Node manager starting
2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 11
2012-09-29 15:19:47,774 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting
2012-09-29 15:19:47,776 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)] Reloading configuration file:conf/test.conf
2012-09-29 15:19:47,780 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] Added sinks: hdfs-sink Agent: agent_foo
2012-09-29 15:19:47,789 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)] Post-validation flume configuration contains configuration for agents: [agent_foo]
2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)] Creating channels
2012-09-29 15:19:47,858 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)] Monitoried counter group for type: CHANNEL, name: mem-channel-1, registered successfully.
2012-09-29 15:19:47,859 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)] created channel mem-channel-1
2012-09-29 15:19:47,879 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)] Creating instance of sink: hdfs-sink, type: hdfs
There is no more output after the last line.
Upgrade hadoop enough.
------------------ Original ------------------
From: "Lichen"<[EMAIL PROTECTED]>;
Date: Sat, Sep 29, 2012 03:57 PM
To: "[EMAIL PROTECTED]"<[EMAIL PROTECTED]>;
Subject: Unable to use HDFS sink
Hi. I’m using flume 1.2 with hadoop-0.20.2 but unable to use HDFS sink. Flume stuck during creating HDFS sink, which is OK using file-roll or logger sinks. I’ve included $HADOOP_HOME and $HADOOP_HOME/lib in FLUME_CLASSPATH. Is it because my hadoop version not compatible?
Configuration file:
agent_foo.sources = netcat-source
agent_foo.sinks = hdfs-sink
agent_foo.channels = mem-channel-1
# properties of netcat-source
agent_foo.sources.netcat-source.channels = mem-channel-1
agent_foo.sources.netcat-source.type = netcat
agent_foo.sources.netcat-source.bind = localhost
agent_foo.sources.netcat-source.port = 44444
# properties of hdfs-sink
agent_foo.sinks.hdfs-sink.channel = mem-channel-1
agent_foo.sinks.hdfs-sink.type = hdfs
agent_foo.sinks.hdfs-sink.hdfs.path = hdfs://hadoop0:9000/flume/
# properties of mem-channel-1
agent_foo.channels.mem-channel-1.type = memory
agent_foo.channels.mem-channel-1.capacity = 1000
agent_foo.channels.mem-channel-1.transactionCapacity = 100
Log:
$ bin/flume-ng agent -c conf/ -f conf/test.conf -n agent_foo
2012-09-29 15:19:47,765 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1
2012-09-29 15:19:47,768 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent_foo
2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] Node manager starting
2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 11
2012-09-29 15:19:47,774 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting
2012-09-29 15:19:47,776 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)] Reloading configuration file:conf/test.conf
2012-09-29 15:19:47,780 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] Added sinks: hdfs-sink Agent: agent_foo
2012-09-29 15:19:47,789 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hdfs-sink
2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)] Post-validation flume configuration contains configuration for agents: [agent_foo]
2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)] Creating channels
2012-09-29 15:19:47,858 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)] Monitoried counter group for type: CHANNEL, name: mem-channel-1, registered successfully.
2012-09-29 15:19:47,859 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)] created channel mem-channel-1
2012-09-29 15:19:47,879 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)] Creating instance of sink: hdfs-sink, type: hdfs
There is no more output after the last line.
+
Yanzhi.liu 2012-09-29, 08:13
