Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> A customer use case


+
Emile Kao 2012-12-04, 10:04
+
Nitin Pawar 2012-12-04, 10:21
Copy link to this message
-
Re: A customer use case
Hi Emile,

On Tue, Dec 4, 2012 at 2:04 AM, Emile Kao <[EMAIL PROTECTED]> wrote:
>
> 1. Which is the best way to implement such a scenario using Flume/ Hadoop?
>

You could use the file spooling client / source to stream these files back
in the latest trunk and upcoming Flume 1.3.0 builds, along with hdfs sink.

2. The customer would like to keep the log files in thier original state
> (file name, size, etc..). Is it practicable using Flume?
>

Not recommended. Flume is an event streaming system, not a file copying
mechanism. If you want to do that, just use some scripts with hadoop fs
-put instead of Flume. Flume provides a bunch of stream-oriented features
on top of its event streaming architecture, such as data enrichment
capabilities, event routing, and configurable file rolling on HDFS, to name
a few.

Regards,
Mike
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB