Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Configure Flume HDFS Sink


Copy link to this message
-
Configure Flume HDFS Sink
I have configured Flume with HDFS Sink and I am facing an issue, no matter
which port I run the NameNode, I still get an HDFS IO Error

17 Jul 2013 22:09:12,508 INFO  [hdfs-VisitSink-call-runner-4]
(org.apache.flume.sink.hdfs.BucketWriter.doOpen:208)  - Creating
/home/cloudera/btbridge/data/visit/export.1374113279142.txt.tmp

17 Jul 2013 22:09:12,532 WARN
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.hdfs.HDFSEventSink.process:456)  - HDFS IO error

java.io.IOException: Failed on local exception: java.io.IOException: Broken
pipe; Host Details : local host is: "localhost.localdomain/127.0.0.1";
destination host is: "localhost.localdomain":8020;
I have tried the following in flume-conf.properties

agent1.sinks.PurePathSink.hdfs.path = /home/cloudera/btbridge/data/pp

or

agent1.sinks.PurePathSink.hdfs.path hdfs://localhost:8020/home/cloudera/btbridge/data/pp

or

agent1.sinks.PurePathSink.hdfs.path hdfs://localhost:9000/home/cloudera/btbridge/data/pp

etc

Any pointers on what is missing. Is this a flume issue or Hadoop issue
Thanks,

Rajesh
+
Rajesh Jain 2013-07-19, 16:14
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB