Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Persist Queue On HDFS


Copy link to this message
-
Re: Persist Queue On HDFS
Joe Stein 2014-01-14, 22:15
There is also hadoop contrib producer and consumer
https://github.com/apache/kafka/tree/0.8/contrib for hdfs

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Tue, Jan 14, 2014 at 4:16 PM, Blender Bl <[EMAIL PROTECTED]> wrote:

>
> Hi,
>
>
> My team trying to implement lambda architecture.
> We need to stream all our new data though Kafka to storm, and HDFS.
>
>
> As i see it were are two options:
>
> Using Camus - not very efficent
> Streaming via Storm - not very efficent
>
> Is it possible to persist the queue's files over the HDFS (with
> short-circuit setting switched on) instead of the local filesystem?
>
>
> It suppose to have similar performance to the local file system, doesn't
> it?
>