Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka producer sends messages efficiency


Copy link to this message
-
Re: Kafka producer sends messages efficiency
Have you tried compression? How powerful is the Kafka broker?

Philip

On Mon, May 6, 2013 at 2:48 PM, Sining Ma <[EMAIL PROTECTED]> wrote:
> Hi,
> I tried some simple performance tests for kafka producer.
> I read 1000 lines of text from one file into memory and write these data to a file, regular socket and using kafka producer.
>
>
> The results is as follows:
> When I use BufferedOutputStream writing the data to a file, it takes 17 milliseconds to write 1000 lines.
>
>                 FileOutputStream fos = new FileOutputStream(filename, true);
>                 BufferedOutputStream bufferdOutput = new BufferedOutputStream(fos);
>
>                 long start = System.currentTimeMillis();
>                 for (String line2: strList) {
>                         bufferdOutput.write(line2.getBytes());
>                 }
>                 long end = System.currentTimeMillis();
>
>
>
> When I use BufferedOutputStream writing the data to a regular socket, it takes 21 milliseconds to write 1000 lines.
>         os = new BufferedOutputStream(new DataOutputStream(socket.getOutputStream()));
>
>                 long start = System.currentTimeMillis();
>                 for (String line: strList) {
>                         os.write(line.getBytes());
>                 }
>                 long end = System.currentTimeMillis();
>
>
>
> At last, I use kafka producer to send the data. My current configuration is
> producer.type=async
>
> max.message.size=1000000
> queue.time=30000
> queue.size=1000
> batch.size=100
>
> It takes 25 milliseconds to write 1000 lines. But if I use sync send, it takes 84 milliseconds.
>
>
> My question is that is there any method that I can reduce the time and improve the efficiency for sending messages through kafka producer.
>
>
>
> -- Regards
>    Sining Ma
>
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB