Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> A backup sink


+
Anat Rozenzon 2013-08-11, 07:49
Copy link to this message
-
Re: A backup sink
You would want to replicate events to two channels and have something
like a file sink pull from the second channel. You may want to look at
http://flume.apache.org/FlumeUserGuide.html#flume-channel-selectors

As to using headers to determine path, you use the hdfs sink but point
it to the local filesystem. The HDFS sink allows header substitution in
the path.

On 08/11/2013 04:49 PM, Anat Rozenzon wrote:
> Hi,
>
> I'm looking for a way to create a sink that will simply 'backup' the
> files I'm sending through an Avro Source.
> I have in the headers the file name, host name etc.
> Is there a way to do store each line in a file according to these headers?
>
> For example store events in a file called
> orig_hostname.orig_filename.timestamp, e.g.:
> myhost.messages.log.1.135678263178
>
> Thanks
> Anat
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB