Kafka, mail # user - Kafka producer sends messages efficiency - 2013-05-06, 21:48
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Kafka producer sends messages efficiency
Hi,
I tried some simple performance tests for kafka producer.
I read 1000 lines of text from one file into memory and write these data to a file, regular socket and using kafka producer.
The results is as follows:
When I use BufferedOutputStream writing the data to a file, it takes 17 milliseconds to write 1000 lines.

FileOutputStream fos = new FileOutputStream(filename, true);
BufferedOutputStream bufferdOutput = new BufferedOutputStream(fos);

long start = System.currentTimeMillis();
for (String line2: strList) {
bufferdOutput.write(line2.getBytes());
}
long end = System.currentTimeMillis();

When I use BufferedOutputStream writing the data to a regular socket, it takes 21 milliseconds to write 1000 lines.
os = new BufferedOutputStream(new DataOutputStream(socket.getOutputStream()));

long start = System.currentTimeMillis();
for (String line: strList) {
os.write(line.getBytes());
}
long end = System.currentTimeMillis();

At last, I use kafka producer to send the data. My current configuration is
producer.type=async

max.message.size=1000000
queue.time=30000
queue.size=1000
batch.size=100

It takes 25 milliseconds to write 1000 lines. But if I use sync send, it takes 84 milliseconds.
My question is that is there any method that I can reduce the time and improve the efficiency for sending messages through kafka producer.

   Sining Ma

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB