Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Time difference between message fetch and message send is very high (~900-1200ms)


Copy link to this message
-
Re: Time difference between message fetch and message send is very high (~900-1200ms)
Kishore,

The end to end latency will be much lower in 08. For 07, you can try to
tune the producer batch size so it sends data quicker. You can also reduce
the log.flush.interval on the broker but that will drive up iops.

Thanks,
Neha
On May 16, 2013 11:39 PM, "Kishore V. Kopalle" <[EMAIL PROTECTED]>
wrote:

> Hi Frank,
>
> Here are the explicitly set properties:
> Consumer:
>     Properties props = new Properties();
>     props.put("zk.connect", KafkaProperties.zkConnect);
>     props.put("groupid", KafkaProperties.groupId);
>     props.put("zk.sessiontimeout.ms", "400");
>     props.put("zk.synctime.ms", "200");
>     props.put("autocommit.interval.ms", "1000");
>
> Producer:
>     props.put("serializer.class", "kafka.serializer.StringEncoder");
>     props.put("zk.connect", "localhost:2181");
>
> Everything else is out of box except for log.default.flush.interval.ms=1
> in
> server.properties which is set according to Stone's suggestion.
>
>
> Regards,
> Kishore
>
> On Fri, May 17, 2013 at 11:48 AM, Francis Dallaire <
> [EMAIL PROTECTED]> wrote:
>
> > What are the settings you have set on the producer ... is there any sync
> > ack ? If so which type ?
> >
> > In any case, it would be useful to know the parameters you set on the
> > consumer and producer.
> >
> > Thanks
> >
> > Frank
> >
> >
> > Sent from Samsung Mobile
> >
> >
> >
> > -------- Original message --------
> > From: "Kishore V. Kopalle" <[EMAIL PROTECTED]>
> > Date: 05-17-2013 2:11 AM (GMT-05:00)
> > To: [EMAIL PROTECTED],Francis Dallaire <
> [EMAIL PROTECTED]>
> > Subject: Re: Time difference between message fetch and message send is
> > very high (~900-1200ms)
> >
> >
> > Hi Frank,
> >
> > Yes, they are on the same machine. Zoo Keeper, Kafka server and the
> > KafkaConsumerProducerDemo (with one producer and one consumer)  are
> running
> > on the same computer.
> >
> > Regards,
> > Kishore
> >
> > On Fri, May 17, 2013 at 11:38 AM, Francis Dallaire <
> > [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
> wrote:
> > Quick question. Are the producer and consumer on the same machine ? If
> > not, are you accounting for any difference in the time  setting between
> > producer and consumer ?
> >
> > Frank
> >
> > Sent from Samsung Mobile
> >
> >
> >
> > -------- Original message --------
> > From: Kishore Kopalle <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
> > Date: 05-17-2013 1:41 AM (GMT-05:00)
> > To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
> > Subject: Time difference between message fetch and message send is very
> > high (~900-1200ms)
> >
> >
> > Hello All,
> >
> > In my earlier mail, I am sorry I did not give lot of details on my setup
> > where I am measuring performance between producer and  consumer. Here are
> > the details:
> >
> > - I am using Kafka version 0.7 downloaded from Apache web site.
> > - I am using a modified version of KafkaConsumerProducerDemo example that
> > comes with  the distribution. I have made the following modifications:
> >
> > - In Producer java file:
> >
> >     int messageNo = 1;
> >     SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd
> > HH:mm:ss.SSS") ;
> >
> >     while(true)
> >     {
> >       String messageStr = new String("Message_" + messageNo);
> >       producer.send(new ProducerData<Integer, String>(topic,
> >         "Message produced at " + dateFormat.format(new Date())
> >         + ": " + messageStr));
> >       messageNo++;
> >     }
> >
> >
> > - In Consumer.java file:
> >
> >     SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd
> > HH:mm:ss.SSS") ;
> >
> >     while(it.hasNext())
> >       System.out.println("Message received at " + dateFormat.format(new
> > Date()) + ": " + ExampleUtils.getMessage(it.next().message()));
> >
> >
> > Based on the messages on the console from consumer, I am getting values
> > ranging from 900ms to 1200ms per message for the time difference between
> > received message time and the timestamp in the message itself.