Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Closing socket connections during extended message delivery


Copy link to this message
-
Re: Closing socket connections during extended message delivery
Neha Narkhede 2013-05-21, 15:44
Yes. This was added to get around the VIP hard killing connections that are
idle. If your producer doesn't use a VIP you can set this high enough so it
rarely reconnects.

Thanks,
Neha
On May 21, 2013 8:13 AM, "Jason Weiss" <[EMAIL PROTECTED]> wrote:

> Ahh - I see, thank you.
>
> Would you describe this as a "tunable" parameter to achieve higher
> performance? For example, is it in a producer's interest to bump this
> figure up to say 250,000 from 30,000?
>
> Thanks again for the prompt response!
>
> Jason
>
>
> On 5/21/13 11:07 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
>
> >In 0.7, each producer reconnects to the broker periodically. Take a look
> >at
> >reconnect.interval in http://kafka.apache.org/07/configuration.html
> >
> >Thanks,
> >
> >Jun
> >
> >
> >On Tue, May 21, 2013 at 7:35 AM, Jason Weiss <[EMAIL PROTECTED]>
> >wrote:
> >
> >> I've made some good progress on tuning Kafka for our needs, reaching
> >>about
> >> 40,000 events per second in our environment. One thing that I noticed in
> >> all of the Kafka server logs is an inordinate amount (I think?) of
> >>Closing
> >> socket connection INFO statements.
> >>
> >> I realize they are INFO, so I assume they are expected. Can someone
> >>point
> >> me to a resource or provide a couple sentence overview on the socket
> >> strategy here? I'm curious why 6 producer clients, each using 2 threads,
> >> are opening and closing sockets so much. That's an expensive operation,
> >>and
> >> surely it affects performance since these connections are closing
> >>anywhere
> >> from 2 seconds to 30 seconds (seemingly randomly).
> >>
> >> Thanks!
> >>
> >> Jason
> >>
> >>
> >>
> >> [2013-05-21 14:31:07,726] INFO Closing socket connection to
> >>/10.10.1.133.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:08,583] INFO Closing socket connection to
> >>/10.10.1.132.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:08,930] INFO Closing socket connection to
> >>/10.10.1.130.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:09,599] INFO Closing socket connection to
> >>/10.10.1.133.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:10,310] INFO Closing socket connection to /10.10.1.67
> .
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:10,436] INFO Closing socket connection to
> >>/10.10.1.131.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:11,165] INFO Closing socket connection to
> >>/10.10.1.134.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:13,260] INFO Closing socket connection to /10.10.1.67
> .
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:13,591] INFO Closing socket connection to
> >>/10.10.1.134.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:20,988] INFO Closing socket connection to
> >>/10.10.1.130.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:22,361] INFO Closing socket connection to
> >>/10.10.1.132.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:22,459] INFO Closing socket connection to
> >>/10.10.1.133.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:23,915] INFO Closing socket connection to
> >>/10.10.1.131.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:24,068] INFO Closing socket connection to
> >>/10.10.1.132.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:24,947] INFO Closing socket connection to
> >>/10.10.1.130.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:26,168] INFO Closing socket connection to
> >>/10.10.1.131.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:26,534] INFO Closing socket connection to
> >>/10.10.1.133.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:26,536] INFO Closing socket connection to /10.10.1.67
> .
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:28,251] INFO Closing socket connection to
> >>/10.10.1.134.
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:30,734] INFO Closing socket connection to /10.10.1.67
> .
> >> (kafka.network.Processor)
> >> [2013-05-21 14:31:31,322] INFO Closing socket connection to
> >>/10.10.1.134.