Hi,
  We have a job that writes many small files (using MultipleOutputFormat)
and its exceeding the 4000 xcievers that we have configured. What is the
effect on the cluster of increasing this count to some higher number?
Many thanks,
   Martin

PD: Hbase is also running on the cluster.

4000 xcievers is a lot.

I'm wondering if there's a correlation between the number of xcievers and ulimit -n. Should they be configured on a 1 to 1 ratio?

-Mike
 
> Date: Thu, 23 Sep 2010 08:04:40 -0400
> Subject: Xcievers Load
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
>
> Hi,
>   We have a job that writes many small files (using MultipleOutputFormat)
> and its exceeding the 4000 xcievers that we have configured. What is the
> effect on the cluster of increasing this count to some higher number?
> Many thanks,
>    Martin
>
> PD: Hbase is also running on the cluster.
     

On Thu, Sep 23, 2010 at 7:05 AM, Michael Segel
<[EMAIL PROTECTED]> wrote:
>
>
> 4000 xcievers is a lot.
>
> I'm wondering if there's a correlation between the number of xcievers and ulimit -n. Should they be configured on a 1 to 1 ratio?
>

2:1 ratio of file descriptors to xceivers. 4000 xceivers is quite
normal on a heavily loaded HBase cluster in my experience.

The cost is the RAM of the extra threads, but there's not much you can
do about it, given the current design of the datanode.

-Todd

> -Mike
>
>> Date: Thu, 23 Sep 2010 08:04:40 -0400
>> Subject: Xcievers Load
>> From: [EMAIL PROTECTED]
>> To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
>>
>> Hi,
>>   We have a job that writes many small files (using MultipleOutputFormat)
>> and its exceeding the 4000 xcievers that we have configured. What is the
>> effect on the cluster of increasing this count to some higher number?
>> Many thanks,
>>    Martin
>>
>> PD: Hbase is also running on the cluster.
>

--
Todd Lipcon
Software Engineer, Cloudera

> From: Todd Lipcon <[EMAIL PROTECTED]>
[...]
> > 4000 xcievers is a lot.
>
> 2:1 ratio of file descriptors to xceivers. 4000 xceivers is
> quite normal on a heavily loaded HBase cluster in my experience.

We run with 10K xceivers...

The problem is the pain is not quite high enough to devote months to making what amounts to a new DataNode; though it is high. We'll reach a tipping point when someone sets up a 1000+ node HBase cluster I expect.

Best regards,

    - Andy

Thanks everyone, we're currently testing with 10K and no issues so far.

On Thu, Sep 23, 2010 at 2:57 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:

> > From: Todd Lipcon <[EMAIL PROTECTED]>
> [...]
> > > 4000 xcievers is a lot.
> >
> > 2:1 ratio of file descriptors to xceivers. 4000 xceivers is
> > quite normal on a heavily loaded HBase cluster in my experience.
>
> We run with 10K xceivers...
>
> The problem is the pain is not quite high enough to devote months to making
> what amounts to a new DataNode; though it is high. We'll reach a tipping
> point when someone sets up a 1000+ node HBase cluster I expect.
>
> Best regards,
>
>    - Andy
>
>
>
>
>
>

On Sep 23, 2010, at 11:57 AM, Andrew Purtell wrote:

>> From: Todd Lipcon <[EMAIL PROTECTED]>
> [...]
>>> 4000 xcievers is a lot.
>>
>> 2:1 ratio of file descriptors to xceivers. 4000 xceivers is
>> quite normal on a heavily loaded HBase cluster in my experience.
>
> We run with 10K xceivers...
>
> The problem is the pain is not quite high enough to devote months to making what amounts to a new DataNode; though it is high. We'll reach a tipping point when someone sets up a 1000+ node HBase cluster I expect.

I hope if/when the datanode is rewritten, they spell check the parameters.