Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Unable to use HDFS sink


+
Lichen 2012-09-29, 07:57
+
Brock Noland 2012-09-29, 14:18
+
Mike Percy 2012-09-30, 02:49
+
Lichen 2012-10-08, 07:37
Copy link to this message
-
Re: Unable to use HDFS sink
Chen if you can send jstack output that could help. Or do: sudo kill -3
<pid> on Flume and send the output from the Flume stdout, which is the same
thing.

Regards
Mike
On Mon, Oct 8, 2012 at 12:37 AM, Lichen <[EMAIL PROTECTED]> wrote:

>  Hi Mike,****
>
> ** **
>
> I’ve tried hadoop-1.0.3 which doesn’t work as before. Nothing happens after
> ****
>
> [INFO -
> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)]
> Creating instance of sink: hdfs-sink, type: hdfs****
>
> I can execute $HADOOP_HOME/bin/hadoop classpath. Don’t know why the HDFS
> sink is not working.****
>
> ** **
>
> Chen****
>
> ** **
>
> On Sat, Sep 30, 2012 at 10:49 AM, Mike Percy <[EMAIL PROTECTED]> wrote::**
> **
>
> ** **
>
> Hi,****
>
> Don't use vanilla hadoop 0.20.2 ... you will not get durability.****
>
> ** **
>
> You must use hadoop 0.20.205, hadoop 1.0.x, hadoop 2.0.x, or CDH3/4 with
> Flume NG.****
>
> ** **
>
> Regards,****
>
> Mike****
>
> ** **
>
> On Sat, Sep 29, 2012 at 12:57 AM, Lichen <[EMAIL PROTECTED]> wrote:***
> *
>
> Hi. I’m using flume 1.2 with hadoop-0.20.2 but unable to use HDFS sink.
> Flume stuck during creating HDFS sink, which is OK using file-roll or
> logger sinks. I’ve included $HADOOP_HOME and $HADOOP_HOME/lib in
> FLUME_CLASSPATH. Is it because my hadoop version not compatible?****
>
>  ****
>
> Configuration file:****
>
> agent_foo.sources = netcat-source****
>
> agent_foo.sinks = hdfs-sink****
>
> agent_foo.channels = mem-channel-1****
>
>  ****
>
> # properties of netcat-source****
>
> agent_foo.sources.netcat-source.channels = mem-channel-1****
>
> agent_foo.sources.netcat-source.type = netcat****
>
> agent_foo.sources.netcat-source.bind = localhost****
>
> agent_foo.sources.netcat-source.port = 44444****
>
>  ****
>
> # properties of hdfs-sink****
>
> agent_foo.sinks.hdfs-sink.channel = mem-channel-1****
>
> agent_foo.sinks.hdfs-sink.type = hdfs****
>
> agent_foo.sinks.hdfs-sink.hdfs.path = hdfs://hadoop0:9000/flume/****
>
>  ****
>
> # properties of mem-channel-1****
>
> agent_foo.channels.mem-channel-1.type = memory****
>
> agent_foo.channels.mem-channel-1.capacity = 1000****
>
> agent_foo.channels.mem-channel-1.transactionCapacity = 100****
>
>  ****
>
> Log:****
>
> $ bin/flume-ng agent -c conf/ -f conf/test.conf -n agent_foo****
>
> 2012-09-29 15:19:47,765 (main) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 1****
>
> 2012-09-29 15:19:47,768 (main) [INFO -
> org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node
> starting - agent_foo****
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)]
> Node manager starting****
>
> 2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 11****
>
> 2012-09-29 15:19:47,774 (lifecycleSupervisor-1-2) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
> Configuration provider starting****
>
> 2012-09-29 15:19:47,776 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)]
> Reloading configuration file:conf/test.conf****
>
> 2012-09-29 15:19:47,780 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
> Processing:hdfs-sink****
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
> Processing:hdfs-sink****
>
> 2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)]
> Added sinks: hdfs-sink Agent: agent_foo****
+
Lichen 2012-10-09, 03:48
+
Mike Percy 2012-10-12, 00:47
+
Hari Shreedharan 2012-09-30, 03:38
+
Yanzhi.liu 2012-09-29, 08:13
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB