Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Persist Queue On HDFS

Copy link to this message
Re: Persist Queue On HDFS
There is also hadoop contrib producer and consumer
https://github.com/apache/kafka/tree/0.8/contrib for hdfs

 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
On Tue, Jan 14, 2014 at 4:16 PM, Blender Bl <[EMAIL PROTECTED]> wrote:

> Hi,
> My team trying to implement lambda architecture.
> We need to stream all our new data though Kafka to storm, and HDFS.
> As i see it were are two options:
> Using Camus - not very efficent
> Streaming via Storm - not very efficent
> Is it possible to persist the queue's files over the HDFS (with
> short-circuit setting switched on) instead of the local filesystem?
> It suppose to have similar performance to the local file system, doesn't
> it?