Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Writing to HDFS from multiple HDFS agents (separate machines)


+
Gary Malouf 2013-03-14, 21:54
+
Mohammad Tariq 2013-03-14, 22:00
+
Seshu V 2013-03-15, 21:20
+
Paul Chavez 2013-03-14, 22:31
Copy link to this message
-
Re: Writing to HDFS from multiple HDFS agents (separate machines)
Gary Malouf 2013-03-14, 22:34
To be clear, I am referring to the segregating of data from different flume
sinks as opposed to the original source of the event.  Having said that, it
sounds like your approach is the easiest.

-Gary
On Thu, Mar 14, 2013 at 5:54 PM, Gary Malouf <[EMAIL PROTECTED]> wrote:

> Hi guys,
>
> I'm new to flume (hdfs for that metter), using the version packaged with
> CDH4 (1.3.0) and was wondering how others are maintaining different file
> names being written to per HDFS sink.
>
> My initial thought is to create a separate sub-directory in hdfs for each
> sink - though I feel like the better way is to somehow prefix each file
> with a unique sink id.  Are there any patterns that others are following
> for this?
>
> -Gary
>
+
Mike Percy 2013-03-15, 01:46
+
Gary Malouf 2013-03-15, 02:30
+
Gary Malouf 2013-03-15, 02:42
+
Mike Percy 2013-03-15, 20:43
+
Paul Chavez 2013-03-15, 03:30