Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> flume 1.4.0 avro source/sink with hdfs sink configuration - no hdfs files created


+
Suhas Satish 2013-09-12, 02:55
+
Suhas Satish 2013-09-12, 03:08
+
Suhas Satish 2013-09-12, 03:16
+
Babu, Prashanth 2013-09-12, 04:15
+
Suhas Satish 2013-09-12, 17:34
+
Paul Chavez 2013-09-12, 17:48
+
Suhas Satish 2013-09-12, 17:57
+
Paul Chavez 2013-09-12, 18:07
Copy link to this message
-
RE: flume 1.4.0 avro source/sink with hdfs sink configuration - no hdfs files created
Yes, it would be better for you to create 2 configurations one each for agent and collector.

Run the following commands for starting the Flume instances in 2 different command shells.
../bin/flume-ng agent --conf ./  -f flume-avro.conf -Dflume.root.logger=DEBUG,LOGFILE -n agent &

../bin/flume-ng agent --conf ./  -f flume-avro.conf -Dflume.root.logger=DEBUG,LOGFILE -n collector &

Regards,
Prashanth.

From: Paul Chavez [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 12, 2013 11:37 PM
To: [EMAIL PROTECTED]
Subject: RE: flume 1.4.0 avro source/sink with hdfs sink configuration - no hdfs files created

I think you can do this on one node, but you will need to run two instances of flume, each with a different agent name.

Paul

From: Suhas Satish [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 12, 2013 10:58 AM
To: [EMAIL PROTECTED]
Subject: Re: flume 1.4.0 avro source/sink with hdfs sink configuration - no hdfs files created

Thanks. Yes I was trying to set it up on a single node. If it cannot be done, I can go to 2 different nodes, but that would add additional complexities which I'd like to avoid if possible.

My original intent was to test if the Avro Source/Avro Sink interface can work with SSL enabled (hence the extra hop) and if it can, can it use the ssl_keystore and ssl_truststore already available from a secure hadoop cluster.

On Thu, Sep 12, 2013 at 10:48 AM, Paul Chavez <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
et this all up on a single node? If so, why are you adding in an extra Avro hop?

In practice this setup should be on two nodes, one acting as the 'agent' with the exec sour

Cheers,
Suhas.

______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data.  If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
+
Mahadevappa, Shobha 2013-09-13, 07:24