Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Writing to HDFS from multiple HDFS agents (separate machines)


+
Gary Malouf 2013-03-14, 21:54
+
Mohammad Tariq 2013-03-14, 22:00
+
Seshu V 2013-03-15, 21:20
+
Paul Chavez 2013-03-14, 22:31
Copy link to this message
-
Re: Writing to HDFS from multiple HDFS agents (separate machines)
To be clear, I am referring to the segregating of data from different flume
sinks as opposed to the original source of the event.  Having said that, it
sounds like your approach is the easiest.

-Gary
On Thu, Mar 14, 2013 at 5:54 PM, Gary Malouf <[EMAIL PROTECTED]> wrote:

> Hi guys,
>
> I'm new to flume (hdfs for that metter), using the version packaged with
> CDH4 (1.3.0) and was wondering how others are maintaining different file
> names being written to per HDFS sink.
>
> My initial thought is to create a separate sub-directory in hdfs for each
> sink - though I feel like the better way is to somehow prefix each file
> with a unique sink id.  Are there any patterns that others are following
> for this?
>
> -Gary
>
+
Mike Percy 2013-03-15, 01:46
+
Gary Malouf 2013-03-15, 02:30
+
Gary Malouf 2013-03-15, 02:42
+
Mike Percy 2013-03-15, 20:43
+
Paul Chavez 2013-03-15, 03:30
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB