Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Unable to use HDFS sink


+
Lichen 2012-09-29, 07:57
+
Brock Noland 2012-09-29, 14:18
Copy link to this message
-
Re: Unable to use HDFS sink
Hi,
Don't use vanilla hadoop 0.20.2 ... you will not get durability.

You must use hadoop 0.20.205, hadoop 1.0.x, hadoop 2.0.x, or CDH3/4 with
Flume NG.

Regards,
Mike

On Sat, Sep 29, 2012 at 12:57 AM, Lichen <[EMAIL PROTECTED]> wrote:

>  Hi. I’m using flume 1.2 with hadoop-0.20.2 but unable to use HDFS sink.
> Flume stuck during creating HDFS sink, which is OK using file-roll or
> logger sinks. I’ve included $HADOOP_HOME and $HADOOP_HOME/lib in
> FLUME_CLASSPATH. Is it because my hadoop version not compatible?****
>
> ** **
>
> Configuration file:****
>
> agent_foo.sources = netcat-source****
>
> agent_foo.sinks = hdfs-sink****
>
> agent_foo.channels = mem-channel-1****
>
> ** **
>
> # properties of netcat-source****
>
> agent_foo.sources.netcat-source.channels = mem-channel-1****
>
> agent_foo.sources.netcat-source.type = netcat****
>
> agent_foo.sources.netcat-source.bind = localhost****
>
> agent_foo.sources.netcat-source.port = 44444****
>
> ** **
>
> # properties of hdfs-sink****
>
> agent_foo.sinks.hdfs-sink.channel = mem-channel-1****
>
> agent_foo.sinks.hdfs-sink.type = hdfs****
>
> agent_foo.sinks.hdfs-sink.hdfs.path = hdfs://hadoop0:9000/flume/****
>
> ** **
>
> # properties of mem-channel-1****
>
> agent_foo.channels.mem-channel-1.type = memory****
>
> agent_foo.channels.mem-channel-1.capacity = 1000****
>
> agent_foo.channels.mem-channel-1.transactionCapacity = 100****
>
> ** **
>
> Log:****
>
> $ bin/flume-ng agent -c conf/ -f conf/test.conf -n agent_foo****
>
> 2012-09-29 15:19:47,765 (main) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 1****
>
> 2012-09-29 15:19:47,768 (main) [INFO -
> org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node
> starting - agent_foo****
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)]
> Node manager starting****
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 11****
>
> 2012-09-29 15:19:47,774 (lifecycleSupervisor-1-2) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
> Configuration provider starting****
>
> 2012-09-29 15:19:47,776 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)]
> Reloading configuration file:conf/test.conf****
>
> 2012-09-29 15:19:47,780 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
> Processing:hdfs-sink****
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
> Processing:hdfs-sink****
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)]
> Added sinks: hdfs-sink Agent: agent_foo****
>
> 2012-09-29 15:19:47,789 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
> Processing:hdfs-sink****
>
> 2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)]
> Post-validation flume configuration contains configuration  for agents:
> [agent_foo]****
>
> 2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)]
> Creating channels****
>
> 2012-09-29 15:19:47,858 (conf-file-poller-0) [INFO -
> org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)]
+
Lichen 2012-10-08, 07:37
+
Mike Percy 2012-10-09, 02:58
+
Lichen 2012-10-09, 03:48
+
Mike Percy 2012-10-12, 00:47
+
Hari Shreedharan 2012-09-30, 03:38
+
Yanzhi.liu 2012-09-29, 08:13