Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka producer sends messages efficiency


+
Sining Ma 2013-05-06, 21:48
+
Philip OToole 2013-05-06, 22:34
Copy link to this message
-
Re: Kafka producer sends messages efficiency
Also, what's the flush.interval on the broker?

Thanks,

Jun
On Mon, May 6, 2013 at 2:48 PM, Sining Ma <[EMAIL PROTECTED]> wrote:

> Hi,
> I tried some simple performance tests for kafka producer.
> I read 1000 lines of text from one file into memory and write these data
> to a file, regular socket and using kafka producer.
>
>
> The results is as follows:
> When I use BufferedOutputStream writing the data to a file, it takes 17
> milliseconds to write 1000 lines.
>
>                 FileOutputStream fos = new FileOutputStream(filename,
> true);
>                 BufferedOutputStream bufferdOutput = new
> BufferedOutputStream(fos);
>
>                 long start = System.currentTimeMillis();
>                 for (String line2: strList) {
>                         bufferdOutput.write(line2.getBytes());
>                 }
>                 long end = System.currentTimeMillis();
>
>
>
> When I use BufferedOutputStream writing the data to a regular socket, it
> takes 21 milliseconds to write 1000 lines.
>         os = new BufferedOutputStream(new
> DataOutputStream(socket.getOutputStream()));
>
>                 long start = System.currentTimeMillis();
>                 for (String line: strList) {
>                         os.write(line.getBytes());
>                 }
>                 long end = System.currentTimeMillis();
>
>
>
> At last, I use kafka producer to send the data. My current configuration is
> producer.type=async
>
> max.message.size=1000000
> queue.time=30000
> queue.size=1000
> batch.size=100
>
> It takes 25 milliseconds to write 1000 lines. But if I use sync send, it
> takes 84 milliseconds.
>
>
> My question is that is there any method that I can reduce the time and
> improve the efficiency for sending messages through kafka producer.
>
>
>
> -- Regards
>    Sining Ma
>
>
>