Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Too many open files


Copy link to this message
-
Re: Too many open files
If a client is gone, the broker should automatically close those broken
sockets. Are you using a hardware load balancer?

Thanks,

Jun
On Wed, Sep 25, 2013 at 4:48 PM, Mark <[EMAIL PROTECTED]> wrote:

> FYI if I kill all producers I don't see the number of open files drop. I
> still see all the ESTABLISHED connections.
>
> Is there a broker setting to automatically kill any inactive TCP
> connections?
>
>
> On Sep 25, 2013, at 4:30 PM, Mark <[EMAIL PROTECTED]> wrote:
>
> > Any other ideas?
> >
> > On Sep 25, 2013, at 9:06 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> >> We haven't seen any socket leaks with the java producer. If you have
> lots
> >> of unexplained socket connections in established mode, one possible
> cause
> >> is that the client created new producer instances, but didn't close the
> old
> >> ones.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Wed, Sep 25, 2013 at 6:08 AM, Mark <[EMAIL PROTECTED]>
> wrote:
> >>
> >>> No. We are using the kafka-rb ruby gem producer.
> >>> https://github.com/acrosa/kafka-rb
> >>>
> >>> Now that you asked that question I need to ask. Is there a problem with
> >>> the java producer?
> >>>
> >>> Sent from my iPhone
> >>>
> >>>> On Sep 24, 2013, at 9:01 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >>>>
> >>>> Are you using the java producer client?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Jun
> >>>>
> >>>>
> >>>>> On Tue, Sep 24, 2013 at 5:33 PM, Mark <[EMAIL PROTECTED]>
> >>> wrote:
> >>>>>
> >>>>> Our 0.7.2 Kafka cluster keeps crashing with:
> >>>>>
> >>>>> 2013-09-24 17:21:47,513 -  [kafka-acceptor:Acceptor@153] - Error in
> >>>>> acceptor
> >>>>>      java.io.IOException: Too many open
> >>>>>
> >>>>> The obvious fix is to bump up the number of open files but I'm
> wondering
> >>>>> if there is a leak on the Kafka side and/or our application side. We
> >>>>> currently have the ulimit set to a generous 4096 but obviously we are
> >>>>> hitting this ceiling. What's a recommended value?
> >>>>>
> >>>>> We are running rails and our Unicorn workers are connecting to our
> Kafka
> >>>>> cluster via round-robin load balancing. We have about 1500 workers to
> >>> that
> >>>>> would be 1500 connections right there but they should be split across
> >>> our 3
> >>>>> nodes. Instead Netstat shows thousands of connections that look like
> >>> this:
> >>>>>
> >>>>> tcp        0      0 kafka1.mycompany.:XmlIpcRegSvc ::ffff:
> >>> 10.99.99.1:22503    ESTABLISHED
> >>>>> tcp        0      0 kafka1.mycompany.:XmlIpcRegSvc ::ffff:
> >>> 10.99.99.1:48398    ESTABLISHED
> >>>>> tcp        0      0 kafka1.mycompany.:XmlIpcRegSvc ::ffff:
> >>> 10.99.99.2:29617    ESTABLISHED
> >>>>> tcp        0      0 kafka1.mycompany.:XmlIpcRegSvc ::ffff:
> >>> 10.99.99.1:32444    ESTABLISHED
> >>>>> tcp        0      0 kafka1.mycompany.:XmlIpcRegSvc ::ffff:
> >>> 10.99.99.1:34415    ESTABLISHED
> >>>>> tcp        0      0 kafka1.mycompany.:XmlIpcRegSvc ::ffff:
> >>> 10.99.99.1:56901    ESTABLISHED
> >>>>> tcp        0      0 kafka1.mycompany.:XmlIpcRegSvc ::ffff:
> >>> 10.99.99.2:45349    ESTABLISHED
> >>>>>
> >>>>> Has anyone come across this problem before? Is this a 0.7.2 leak, LB
> >>>>> misconfiguration… ?
> >>>>>
> >>>>> Thanks
> >>>
> >
>
>