Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Too many open files


Copy link to this message
-
Re: Too many open files
Are you using the java or non-java producer? Are you using ZK based,
broker-list based, or VIP based producer?

Thanks,

Jun
On Wed, Sep 25, 2013 at 10:06 PM, Nicolas Berthet
<[EMAIL PROTECTED]>wrote:

> Jun,
>
> I observed similar kind of things recently. (didn't notice before because
> our file limit is huge)
>
> I have a set of brokers in a datacenter, and producers in different data
> centers.
>
> At some point I got disconnections, from the producer perspective I had
> something like 15 connections to the broker. On the other hand on the
> broker side, I observed hundreds of connections from the producer in an
> ESTABLISHED state.
>
> We had some default settings for the socket timeout on the OS level, which
> we reduced hoping it would prevent the issue in the future. I'm not sure if
> the issue is from the broker or OS configuration though. I'm still keeping
> the broker under observation for the time being.
>
> Note that, for clients in the same datacenter, we didn't see this issue,
> the socket count matches on both ends.
>
> Nicolas Berthet
>
> -----Original Message-----
> From: Jun Rao [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, September 26, 2013 12:39 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Too many open files
>
> If a client is gone, the broker should automatically close those broken
> sockets. Are you using a hardware load balancer?
>
> Thanks,
>
> Jun
>
>
> On Wed, Sep 25, 2013 at 4:48 PM, Mark <[EMAIL PROTECTED]> wrote:
>
> > FYI if I kill all producers I don't see the number of open files drop.
> > I still see all the ESTABLISHED connections.
> >
> > Is there a broker setting to automatically kill any inactive TCP
> > connections?
> >
> >
> > On Sep 25, 2013, at 4:30 PM, Mark <[EMAIL PROTECTED]> wrote:
> >
> > > Any other ideas?
> > >
> > > On Sep 25, 2013, at 9:06 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >
> > >> We haven't seen any socket leaks with the java producer. If you
> > >> have
> > lots
> > >> of unexplained socket connections in established mode, one possible
> > cause
> > >> is that the client created new producer instances, but didn't close
> > >> the
> > old
> > >> ones.
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >>
> > >> On Wed, Sep 25, 2013 at 6:08 AM, Mark <[EMAIL PROTECTED]>
> > wrote:
> > >>
> > >>> No. We are using the kafka-rb ruby gem producer.
> > >>> https://github.com/acrosa/kafka-rb
> > >>>
> > >>> Now that you asked that question I need to ask. Is there a problem
> > >>> with the java producer?
> > >>>
> > >>> Sent from my iPhone
> > >>>
> > >>>> On Sep 24, 2013, at 9:01 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >>>>
> > >>>> Are you using the java producer client?
> > >>>>
> > >>>> Thanks,
> > >>>>
> > >>>> Jun
> > >>>>
> > >>>>
> > >>>>> On Tue, Sep 24, 2013 at 5:33 PM, Mark
> > >>>>> <[EMAIL PROTECTED]>
> > >>> wrote:
> > >>>>>
> > >>>>> Our 0.7.2 Kafka cluster keeps crashing with:
> > >>>>>
> > >>>>> 2013-09-24 17:21:47,513 -  [kafka-acceptor:Acceptor@153] - Error
> > >>>>> in acceptor
> > >>>>>      java.io.IOException: Too many open
> > >>>>>
> > >>>>> The obvious fix is to bump up the number of open files but I'm
> > wondering
> > >>>>> if there is a leak on the Kafka side and/or our application
> > >>>>> side. We currently have the ulimit set to a generous 4096 but
> > >>>>> obviously we are hitting this ceiling. What's a recommended value?
> > >>>>>
> > >>>>> We are running rails and our Unicorn workers are connecting to
> > >>>>> our
> > Kafka
> > >>>>> cluster via round-robin load balancing. We have about 1500
> > >>>>> workers to
> > >>> that
> > >>>>> would be 1500 connections right there but they should be split
> > >>>>> across
> > >>> our 3
> > >>>>> nodes. Instead Netstat shows thousands of connections that look
> > >>>>> like
> > >>> this:
> > >>>>>
> > >>>>> tcp        0      0 kafka1.mycompany.:XmlIpcRegSvc ::ffff:
> > >>> 10.99.99.1:22503    ESTABLISHED
> > >>>>> tcp        0      0 kafka1.mycompany.:XmlIpcRegSvc ::ffff: