Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Persist Queue On HDFS


Copy link to this message
-
Re: Persist Queue On HDFS
There is also hadoop contrib producer and consumer
https://github.com/apache/kafka/tree/0.8/contrib for hdfs

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Tue, Jan 14, 2014 at 4:16 PM, Blender Bl <[EMAIL PROTECTED]> wrote:

>
> Hi,
>
>
> My team trying to implement lambda architecture.
> We need to stream all our new data though Kafka to storm, and HDFS.
>
>
> As i see it were are two options:
>
> Using Camus - not very efficent
> Streaming via Storm - not very efficent
>
> Is it possible to persist the queue's files over the HDFS (with
> short-circuit setting switched on) instead of the local filesystem?
>
>
> It suppose to have similar performance to the local file system, doesn't
> it?
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB