Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka producer sends messages efficiency


Copy link to this message
-
Kafka producer sends messages efficiency
Hi,
I tried some simple performance tests for kafka producer.
I read 1000 lines of text from one file into memory and write these data to a file, regular socket and using kafka producer.
The results is as follows:
When I use BufferedOutputStream writing the data to a file, it takes 17 milliseconds to write 1000 lines.

FileOutputStream fos = new FileOutputStream(filename, true);
BufferedOutputStream bufferdOutput = new BufferedOutputStream(fos);

long start = System.currentTimeMillis();
for (String line2: strList) {
bufferdOutput.write(line2.getBytes());
}
long end = System.currentTimeMillis();

When I use BufferedOutputStream writing the data to a regular socket, it takes 21 milliseconds to write 1000 lines.
os = new BufferedOutputStream(new DataOutputStream(socket.getOutputStream()));

long start = System.currentTimeMillis();
for (String line: strList) {
os.write(line.getBytes());
}
long end = System.currentTimeMillis();

At last, I use kafka producer to send the data. My current configuration is
producer.type=async

max.message.size=1000000
queue.time=30000
queue.size=1000
batch.size=100

It takes 25 milliseconds to write 1000 lines. But if I use sync send, it takes 84 milliseconds.
My question is that is there any method that I can reduce the time and improve the efficiency for sending messages through kafka producer.

-- Regards
   Sining Ma

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB