Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Closing socket connections during extended message delivery


+
Jason Weiss 2013-05-21, 14:36
+
Jun Rao 2013-05-21, 15:08
+
Jason Weiss 2013-05-21, 15:13
Copy link to this message
-
Re: Closing socket connections during extended message delivery
Yes. This was added to get around the VIP hard killing connections that are
idle. If your producer doesn't use a VIP you can set this high enough so it
rarely reconnects.

Thanks,
Neha
On May 21, 2013 8:13 AM, "Jason Weiss" <[EMAIL PROTECTED]> wrote:

> Ahh - I see, thank you.
>
> Would you describe this as a "tunable" parameter to achieve higher
> performance? For example, is it in a producer's interest to bump this
> figure up to say 250,000 from 30,000?
>
> Thanks again for the prompt response!
>
> Jason
>
>
> On 5/21/13 11:07 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
>
> >In 0.7, each producer reconnects to the broker periodically. Take a look
> >at
> >reconnect.interval in http://kafka.apache.org/07/configuration.html
> >
> >Thanks,
> >
> >Jun
> >
> >
> >On Tue, May 21, 2013 at 7:35 AM, Jason Weiss <[EMAIL PROTECTED]>
> >wrote:
> >
> >> I've made some good progress on tuning Kafka for our needs, reaching
> >>about
> >> 40,000 events per second in our environment. One thing that I noticed in
> >> all of the Kafka server logs is an inordinate amount (I think?) of
> >>Closing
> >> socket connection INFO statements.
> >>
> >> I realize they are INFO, so I assume they are expected. Can someone
> >>point
> >> me to a resource or provide a couple sentence overview on the socket
> >> strategy here? I'm curious why 6 producer clients, each using 2 threads,
> >> are opening and closing sockets so much. That's an expensive operation,
> >>and
> >> surely it affects performance since these connections are closing
> >>anywhere
> >> from 2 seconds to 30 seconds (seemingly randomly).
> >>
> >> Thanks!
> >>
> >> Jason
> >>
> >>
> >>
> >> [2013-05-21 14:31:07,726] INFO Closing socket connection to
> >>/10.10.1.133.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:08,583] INFO Closing socket connection to
> >>/10.10.1.132.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:08,930] INFO Closing socket connection to
> >>/10.10.1.130.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:09,599] INFO Closing socket connection to
> >>/10.10.1.133.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:10,310] INFO Closing socket connection to /10.10.1.67
> .
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:10,436] INFO Closing socket connection to
> >>/10.10.1.131.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:11,165] INFO Closing socket connection to
> >>/10.10.1.134.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:13,260] INFO Closing socket connection to /10.10.1.67
> .
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:13,591] INFO Closing socket connection to
> >>/10.10.1.134.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:20,988] INFO Closing socket connection to
> >>/10.10.1.130.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:22,361] INFO Closing socket connection to
> >>/10.10.1.132.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:22,459] INFO Closing socket connection to
> >>/10.10.1.133.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:23,915] INFO Closing socket connection to
> >>/10.10.1.131.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:24,068] INFO Closing socket connection to
> >>/10.10.1.132.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:24,947] INFO Closing socket connection to
> >>/10.10.1.130.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:26,168] INFO Closing socket connection to
> >>/10.10.1.131.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:26,534] INFO Closing socket connection to
> >>/10.10.1.133.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:26,536] INFO Closing socket connection to /10.10.1.67
> .
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:28,251] INFO Closing socket connection to
> >>/10.10.1.134.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:30,734] INFO Closing socket connection to /10.10.1.67
> .
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:31,322] INFO Closing socket connection to
> >>/10.10.1.134.

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB