Hi,
Does anyone have any idea about how we can use HDFS with ZooKeeper? More
elaborately if NameNode fails DataNodes should be able to retrieve address
of new NameNode from ZooKeeper.

Thanks,
---------------------------------
Ozcan ILIKHAN
PhD Student, Graduate Research Assistant
Department of Computer Sciences
University of Wisconsin-Madison
http://pages.cs.wisc.edu/~ilikhan

This is a 1M$ question. You could start thinking about this problem by
looking at what "AvatarNode" does:
https://issues.apache.org/jira/browse/HDFS-976

J-D

On Fri, Apr 22, 2011 at 10:17 PM, Ozcan ILIKHAN <[EMAIL PROTECTED]> wrote:
> Hi,
> Does anyone have any idea about how we can use HDFS with ZooKeeper? More
> elaborately if NameNode fails DataNodes should be able to retrieve address
> of new NameNode from ZooKeeper.
>
> Thanks,
> ---------------------------------
> Ozcan ILIKHAN
> PhD Student, Graduate Research Assistant
> Department of Computer Sciences
> University of Wisconsin-Madison
> http://pages.cs.wisc.edu/~ilikhan
>

This is an interesting question.  AvatarNode doesn't use Zookeeper?
Is there a drawback to implementing ZK for the NameNode other than the
potential bugs from a newly engineered feature?

Perhaps this issue is the main one for implementing the NameNode in
ZK?  HDFS-1623

On Fri, Apr 22, 2011 at 10:41 PM, Jean-Daniel Cryans
<[EMAIL PROTECTED]> wrote:
> This is a 1M$ question. You could start thinking about this problem by
> looking at what "AvatarNode" does:
> https://issues.apache.org/jira/browse/HDFS-976
>
> J-D
>
> On Fri, Apr 22, 2011 at 10:17 PM, Ozcan ILIKHAN <[EMAIL PROTECTED]> wrote:
>> Hi,
>> Does anyone have any idea about how we can use HDFS with ZooKeeper? More
>> elaborately if NameNode fails DataNodes should be able to retrieve address
>> of new NameNode from ZooKeeper.
>>
>> Thanks,
>> ---------------------------------
>> Ozcan ILIKHAN
>> PhD Student, Graduate Research Assistant
>> Department of Computer Sciences
>> University of Wisconsin-Madison
>> http://pages.cs.wisc.edu/~ilikhan
>>
>

The AvatarNode does use zookeeper (but since this is not directly related to
Apache HDFS code, if u have more questions, please send it to me directly).

The latest AvatarNode code is in
https://github.com/facebook/hadoop-20-append/tree/master/src/contrib/highavailability/src/java/org/apache/hadoop/hdfs

thanks,
dhruba

On Sat, Apr 23, 2011 at 11:17 AM, Jason Rutherglen <
[EMAIL PROTECTED]> wrote:

> This is an interesting question.  AvatarNode doesn't use Zookeeper?
> Is there a drawback to implementing ZK for the NameNode other than the
> potential bugs from a newly engineered feature?
>
> Perhaps this issue is the main one for implementing the NameNode in
> ZK?  HDFS-1623
>
> On Fri, Apr 22, 2011 at 10:41 PM, Jean-Daniel Cryans
> <[EMAIL PROTECTED]> wrote:
> > This is a 1M$ question. You could start thinking about this problem by
> > looking at what "AvatarNode" does:
> > https://issues.apache.org/jira/browse/HDFS-976
> >
> > J-D
> >
> > On Fri, Apr 22, 2011 at 10:17 PM, Ozcan ILIKHAN <[EMAIL PROTECTED]>
> wrote:
> >> Hi,
> >> Does anyone have any idea about how we can use HDFS with ZooKeeper? More
> >> elaborately if NameNode fails DataNodes should be able to retrieve
> address
> >> of new NameNode from ZooKeeper.
> >>
> >> Thanks,
> >> ---------------------------------
> >> Ozcan ILIKHAN
> >> PhD Student, Graduate Research Assistant
> >> Department of Computer Sciences
> >> University of Wisconsin-Madison
> >> http://pages.cs.wisc.edu/~ilikhan
> >>
> >
>

--
Connect to me at http://www.facebook.com/dhruba

Will the avatar node be part of 0.22?
On Sat, Apr 23, 2011 at 6:29 PM, Dhruba Borthakur <[EMAIL PROTECTED]> wrote:
> The AvatarNode does use zookeeper (but since this is not directly related to
> Apache HDFS code, if u have more questions, please send it to me directly).
> The latest AvatarNode code is in
> https://github.com/facebook/hadoop-20-append/tree/master/src/contrib/highavailability/src/java/org/apache/hadoop/hdfs
> thanks,
> dhruba
>
> On Sat, Apr 23, 2011 at 11:17 AM, Jason Rutherglen
> <[EMAIL PROTECTED]> wrote:
>>
>> This is an interesting question.  AvatarNode doesn't use Zookeeper?
>> Is there a drawback to implementing ZK for the NameNode other than the
>> potential bugs from a newly engineered feature?
>>
>> Perhaps this issue is the main one for implementing the NameNode in
>> ZK?  HDFS-1623
>>
>> On Fri, Apr 22, 2011 at 10:41 PM, Jean-Daniel Cryans
>> <[EMAIL PROTECTED]> wrote:
>> > This is a 1M$ question. You could start thinking about this problem by
>> > looking at what "AvatarNode" does:
>> > https://issues.apache.org/jira/browse/HDFS-976
>> >
>> > J-D
>> >
>> > On Fri, Apr 22, 2011 at 10:17 PM, Ozcan ILIKHAN <[EMAIL PROTECTED]>
>> > wrote:
>> >> Hi,
>> >> Does anyone have any idea about how we can use HDFS with ZooKeeper?
>> >> More
>> >> elaborately if NameNode fails DataNodes should be able to retrieve
>> >> address
>> >> of new NameNode from ZooKeeper.
>> >>
>> >> Thanks,
>> >> ---------------------------------
>> >> Ozcan ILIKHAN
>> >> PhD Student, Graduate Research Assistant
>> >> Department of Computer Sciences
>> >> University of Wisconsin-Madison
>> >> http://pages.cs.wisc.edu/~ilikhan
>> >>
>> >
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>

I have read  Dhruba's blog entry about AvatarNode long ago. If I am not
wrong initially it was not using ZooKeeper. That is way I raised that
question. Also, AvatorNode is build on HDFS 0.20 but my intension is to
utilize BackupNode that comes with HDFS 0.21

Thanks,
Ozcan.

-----Original Message-----
From: Jason Rutherglen
Sent: Saturday, April 23, 2011 1:17 PM
To: [EMAIL PROTECTED]
Subject: Re: HDFS + ZooKeeper

This is an interesting question.  AvatarNode doesn't use Zookeeper?
Is there a drawback to implementing ZK for the NameNode other than the
potential bugs from a newly engineered feature?

Perhaps this issue is the main one for implementing the NameNode in
ZK?  HDFS-1623

On Fri, Apr 22, 2011 at 10:41 PM, Jean-Daniel Cryans
<[EMAIL PROTECTED]> wrote:
> This is a 1M$ question. You could start thinking about this problem by
> looking at what "AvatarNode" does:
> https://issues.apache.org/jira/browse/HDFS-976
>
> J-D
>
> On Fri, Apr 22, 2011 at 10:17 PM, Ozcan ILIKHAN <[EMAIL PROTECTED]>
> wrote:
>> Hi,
>> Does anyone have any idea about how we can use HDFS with ZooKeeper? More
>> elaborately if NameNode fails DataNodes should be able to retrieve
>> address
>> of new NameNode from ZooKeeper.
>>
>> Thanks,
>> ---------------------------------
>> Ozcan ILIKHAN
>> PhD Student, Graduate Research Assistant
>> Department of Computer Sciences
>> University of Wisconsin-Madison
>> http://pages.cs.wisc.edu/~ilikhan
>>
>

Right and AN is not using ZK for the actual NameNode
methods/functions, only for the failover election of the 'backup'
NameNode?

On Sun, Apr 24, 2011 at 4:06 AM, Ozcan ILIKHAN <[EMAIL PROTECTED]> wrote:
> I have read  Dhruba's blog entry about AvatarNode long ago. If I am not
> wrong initially it was not using ZooKeeper. That is way I raised that
> question. Also, AvatorNode is build on HDFS 0.20 but my intension is to
> utilize BackupNode that comes with HDFS 0.21
>
> Thanks,
> Ozcan.
>
> -----Original Message----- From: Jason Rutherglen
> Sent: Saturday, April 23, 2011 1:17 PM
> To: [EMAIL PROTECTED]
> Subject: Re: HDFS + ZooKeeper
>
> This is an interesting question.  AvatarNode doesn't use Zookeeper?
> Is there a drawback to implementing ZK for the NameNode other than the
> potential bugs from a newly engineered feature?
>
> Perhaps this issue is the main one for implementing the NameNode in
> ZK?  HDFS-1623
>
> On Fri, Apr 22, 2011 at 10:41 PM, Jean-Daniel Cryans
> <[EMAIL PROTECTED]> wrote:
>>
>> This is a 1M$ question. You could start thinking about this problem by
>> looking at what "AvatarNode" does:
>> https://issues.apache.org/jira/browse/HDFS-976
>>
>> J-D
>>
>> On Fri, Apr 22, 2011 at 10:17 PM, Ozcan ILIKHAN <[EMAIL PROTECTED]>
>> wrote:
>>>
>>> Hi,
>>> Does anyone have any idea about how we can use HDFS with ZooKeeper? More
>>> elaborately if NameNode fails DataNodes should be able to retrieve
>>> address
>>> of new NameNode from ZooKeeper.
>>>
>>> Thanks,
>>> ---------------------------------
>>> Ozcan ILIKHAN
>>> PhD Student, Graduate Research Assistant
>>> Department of Computer Sciences
>>> University of Wisconsin-Madison
>>> http://pages.cs.wisc.edu/~ilikhan
>>>
>>
>
>

> From: Jason Rutherglen <[EMAIL PROTECTED]>
> Right and AN is not using ZK for the actual NameNode
> methods/functions, only for the failover election of the 'backup'
> NameNode?

What do you mean by "actual NameNode methods/functions"? I believe there was an effort during the development of AvatarNode to use BookKeeper (http://zookeeper.apache.org/doc/r3.3.3/bookkeeperOverview.html) to distribute the edit log. Is this what you mean?

That effort ran into various problems but I don't know the detail.

   - Andy

Andy/others,
 The BookKeeper work hasnt been completed yet. There is ongoing work in

https://issues.apache.org/jira/browse/HDFS-1580

and others.

Hopefully we'll have NN's running on BK soon!

thanks
mahadev

On Sun, Apr 24, 2011 at 10:20 AM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
>> From: Jason Rutherglen <[EMAIL PROTECTED]>
>> Right and AN is not using ZK for the actual NameNode
>> methods/functions, only for the failover election of the 'backup'
>> NameNode?
>
> What do you mean by "actual NameNode methods/functions"? I believe there was an effort during the development of AvatarNode to use BookKeeper (http://zookeeper.apache.org/doc/r3.3.3/bookkeeperOverview.html) to distribute the edit log. Is this what you mean?
>
> That effort ran into various problems but I don't know the detail.
>
>   - Andy
>
>

--
thanks
mahadev
@mahadevkonar