Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Scenarios of Hadoop producers and consumers


Copy link to this message
-
Re: Scenarios of Hadoop producers and consumers
When you need your data streams to be incrementally loaded into hadoop for
offline batch processing and/or ad-hoc querying - some things cannot (or
are expensive to) be computed in real-time. So you have a hadoop job that
consumes kafka stream, potentially formats the data and saves into hdfs.

On 30 October 2012 23:28, Hussein Baghdadi <[EMAIL PROTECTED]> wrote:

>
>
>
>
> Hi,Kafka comes with a support for Hadoop. I'm not sure what does this
> mean.Kafka is a publish-subscribe messaging system. What are some of the
> typical usage of Kafka-support for Hadoop producers and consumers?Well,
> producers are easy to digest. MapReduce job emitting data to Kafka.But what
> about Hadoop consumers?Hadoop is a batching system, not a continuous
> running system (as Storm or Dempsy). Say Kafka gets some data, what will
> happen?Thanks for help and time.
>
--
Michal Haris
Software Engineer

www.visualdna.com | t: +44 (0) 207 734 7033
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB