Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> flume 1.4.0 avro source/sink with hdfs sink configuration - no hdfs files created


Copy link to this message
-
RE: flume 1.4.0 avro source/sink with hdfs sink configuration - no hdfs files created
Are you trying to set this all up on a single node? If so, why are you adding in an extra Avro hop?

In practice this setup should be on two nodes, one acting as the 'agent' with the exec source and avro sink, and the other with an avro source and hdfs sink.

You can use one configuration file, but make sure each node configuration has a different name. Then on each node invoke flume with the proper name.

Hope that helps,
Paul Chavez

From: Suhas Satish [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 12, 2013 10:34 AM
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Cc: dev
Subject: Re: flume 1.4.0 avro source/sink with hdfs sink configuration - no hdfs files created

Yes I tried splitting into agent and collector (with different names) but it was in a single configuration file  flume-avro.conf.

does that mean I need to have 2 separate configuration files, 1 for the agent and 1 for the collector?

If so, is my flume launch command still the same then ? is it

../bin/flume-ng agent --conf ./  -f flume-avro.conf -Dflume.root.logger=DEBUG,LOGFILE -n agent &
or  is it
../bin/flume-ng agent --conf ./  -f flume-avro.conf -Dflume.root.logger=DEBUG,LOGFILE -n collector &
or is it
../bin/flume-ng agent --conf ./  -f flume-avro.conf -Dflume.root.logger=DEBUG,LOGFILE -n agent, collector &

?
On Wed, Sep 11, 2013 at 9:15 PM, Babu, Prashanth <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
d also change agent to something else in the collector conf.
Cheers,
Suhas.