Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Can Hadoop replace the use of MQ b/w processes?


+
Robert Nicholson 2012-08-19, 16:46
+
Michael Segel 2012-08-19, 21:49
+
Lance Norskog 2012-08-20, 00:43
+
Russell Jurney 2012-08-20, 01:31
+
Lance Norskog 2012-08-20, 01:57
+
Ted Dunning 2012-08-20, 02:13
+
Russell Jurney 2012-08-19, 17:27
Copy link to this message
-
Re: Can Hadoop replace the use of MQ b/w processes?
Hi Robert

To add to Russell's answer:

If real-time processing of events is required, you might want to use a
stream-processing system like Apache S4 or Twitter's Storm.

Karthik

On Sun, Aug 19, 2012 at 10:27 AM, Russell Jurney
<[EMAIL PROTECTED]>wrote:

> The model with Hadoop would be to aggregate and write your events to
> The Hadoop Distributed FileSystem, and then process them with
> scheduled batch jobs via Hadoop MapReduce. If your requirements can
> include some latency - then Hadoop can work for you. Depending on your
> processing, you can schedule jobs down to say... every hour, half hour
> or fifteen minutes? I'm not aware or anyone scheduling jobs more
> frequently than that, but they may be. Chime in if you are.
>
> For getting events to HDFS, look at Flume, Kafka and Scribe. For
> processing events, look at Pig, HIVE and Cascading. For scheduling
> jobs look at Oozie and Azkaban.
>
> Russell Jurney http://datasyndrome.com
>
> On Aug 19, 2012, at 9:47 AM, Robert Nicholson
> <[EMAIL PROTECTED]> wrote:
>
> > We have an application or a series of applications that listen to
> incoming feeds they then distribute this data in XML form to a number of
> queues.  Another set of processes listen to these queues and process the
> messages. Order of processing is important in so far as related messages
> need to be processed in sequence hence today all related messages go to the
> same queue and are processed by the same queue consumer.
> >
> > The idea would be replace the use of MQ with some kind of reliable
> distributed dispatch. Does Hadoop provide that?
> >
> >
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB