Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Kafka Broker - In Memory Topics & Messages


Copy link to this message
-
Re: Kafka Broker - In Memory Topics & Messages
Sounds like you want something like zeromq?

http://zguide.zeromq.org/page:all#Divide-and-Conquer

-David

On 4/23/13 2:07 AM, Pankaj Misra wrote:
> Hi All,
>
> I am working on using Kafka for building a highly scalable system. As I understand and have seen, Kafka broker has a very impressive and scalable file handling mechanisms to provide guaranteed delivery. However in one of the scenarios, I am facing a different challenge.
>
> The scenario is such that the message payload is buffered and guaranteed for delivery by an external system, wherein there is no compelling need for guaranteed delivery from Kafka, but there is a need to parallel process the message streams. This made me wonder, if there is some way in Kafka, wherein I can avoid creation of files and instead stream the messages in-memory as they come and still take advantage of Kafka message streams, avoiding the small overhead of file management (avoid some more disk level IOPS).
>
> Would greatly appreciate community's response.
>
> Thanks & Regards
> Pankaj Misra
>
>
> ________________________________
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
>
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB