Hello,

I am currently using curator to manage leader election (which I find really
nice btw), and I have a question:

I am using leaders to manage shards of a database. When a node (=a jvm) is
started, it is told what shards it could manage if it is elected for this
shard. A node can be elected for multiple shards.

The issue I got is that a node typically manages hundreds of shards. Not
sure I understood correctly, but current leader election receipe (as
implemented in curator) results in the creation of hundreds of threads,
which I keep sleeping because actual processing is done by a separate
thread pool.

Is there a way not to use that many threads, given that when a node is
elected for a shard it will never release its leadership?

Thanks

John

This might be better directed to the Curator list (ccing them).

I'm guessing this "thread per leader" is related to a curator pattern?
Using the ZK api directly you should be able to use a single thread to
hold multiple leaderships. Perhaps there's a way to get this behavior
using Curator.

Patrick

On Mon, May 7, 2012 at 5:13 AM, john doe <[EMAIL PROTECTED]> wrote:
> Hello,
>
> I am currently using curator to manage leader election (which I find really
> nice btw), and I have a question:
>
> I am using leaders to manage shards of a database. When a node (=a jvm) is
> started, it is told what shards it could manage if it is elected for this
> shard. A node can be elected for multiple shards.
>
> The issue I got is that a node typically manages hundreds of shards. Not
> sure I understood correctly, but current leader election receipe (as
> implemented in curator) results in the creation of hundreds of threads,
> which I keep sleeping because actual processing is done by a separate
> thread pool.
>
> Is there a way not to use that many threads, given that when a node is
> elected for a shard it will never release its leadership?
>
> Thanks
>
> John

I've decided to add an alternate version of leader selection that doesn't
use threads. It will behave somewhat like a CountDownLatch so I'm calling
it LeaderLatch. I'll report back (only on the Curator list) in the next
day or so when it's available.

-Jordan

On 5/9/12 2:51 PM, "Patrick Hunt" <[EMAIL PROTECTED]> wrote:

>This might be better directed to the Curator list (ccing them).
>
>I'm guessing this "thread per leader" is related to a curator pattern?
>Using the ZK api directly you should be able to use a single thread to
>hold multiple leaderships. Perhaps there's a way to get this behavior
>using Curator.
>
>Patrick
>
>On Mon, May 7, 2012 at 5:13 AM, john doe <[EMAIL PROTECTED]> wrote:
>> Hello,
>>
>> I am currently using curator to manage leader election (which I find
>>really
>> nice btw), and I have a question:
>>
>> I am using leaders to manage shards of a database. When a node (=a jvm)
>>is
>> started, it is told what shards it could manage if it is elected for
>>this
>> shard. A node can be elected for multiple shards.
>>
>> The issue I got is that a node typically manages hundreds of shards. Not
>> sure I understood correctly, but current leader election receipe (as
>> implemented in curator) results in the creation of hundreds of threads,
>> which I keep sleeping because actual processing is done by a separate
>> thread pool.
>>
>> Is there a way not to use that many threads, given that when a node is
>> elected for a shard it will never release its leadership?
>>
>> Thanks
>>
>> John
>