Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # user >> storm and accumulo


Copy link to this message
-
RE: storm and accumulo
I'm trying to connect message queues to a topology that will terminate
in a batchWriter.  Along the way I want to put some bolts in place that
detect certain events/sequences and make the proper asynchronous
notifications.  I'm considering storm for its ability to scale
horizontally because I expect the number of message queues and message
volume to grow considerably.  That being said I'm wondering if I should
build in some scalability into the batchWriters (i.e. the number of
instances of batchWriter) or will it scale by itself as long as I
configure it with the upper end of the volume, like the diagram below
(if it makes it through).

 

 

From: Juan Moreno [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, October 17, 2012 12:16
To: [EMAIL PROTECTED]
Subject: Re: storm and accumulo

 

Hello Bob,

 

We use Accumulo and Storm together in my project and have had good
success with it. The only challenge  of course, is when you care about
order and exactly-once semantics. That said, Storm's transactional
Topologies isn't something easily adapted for Accumulo.

Also, if you care about guaranteeing data processing, be sure to do some
manual acking.
Accumulo is failry record/tuple oriented which works well with storm.
What are you trying to do with Storm and Accumulo exactly?

On Wed, Oct 17, 2012 at 12:55 PM, <[EMAIL PROTECTED]> wrote:

Has anyone done any integration between storm an accumulo?  Primarily
I'm interested in establish some topology from message queues to
batchWriter(s) while performing real-time analytics on the contents.

Bob Thorman
Engineering Fellow
L-3 Communications, ComCept
1700 Science Place
Rockwall, TX 75032
(972) 772-7501 <tel:%28972%29%20772-7501>  work
[EMAIL PROTECTED]
[EMAIL PROTECTED]

 

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB