Hi, guys,

Do we have any plan to enable VIP between NameNode and DataNodes, I mean
multiple NameNodes and we use a VIP stands before the NNs. Then DNs only
need to connect to the VIP. I don't know if it is valuable we have this impl
in our hdsf and if there has any issues?

Any hints are valuable for me. :)

regards
macf

This is a good approach, is used by many to mask the NameNode address
for the DataNodes; and is also good to use while using BackupNode
(Even Facebook does this).

On Wed, Jan 26, 2011 at 7:38 AM, mac fang <[EMAIL PROTECTED]> wrote:
> Hi, guys,
>
> Do we have any plan to enable VIP between NameNode and DataNodes, I mean
> multiple NameNodes and we use a VIP stands before the NNs. Then DNs only
> need to connect to the VIP. I don't know if it is valuable we have this impl
> in our hdsf and if there has any issues?
>
> Any hints are valuable for me. :)
>
> regards
> macf
>

--
Harsh J
www.harshj.com

Thanks, Harsh.

Any issues you see if we use this? I've already done some researches of this
approach. However i am NOT sure if there has some potential risks if we use
VIP (i am NOT a seasoned hadooper :) )

regards
macf

On Wed, Jan 26, 2011 at 10:12 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> This is a good approach, is used by many to mask the NameNode address
> for the DataNodes; and is also good to use while using BackupNode
> (Even Facebook does this).
>
> On Wed, Jan 26, 2011 at 7:38 AM, mac fang <[EMAIL PROTECTED]> wrote:
> > Hi, guys,
> >
> > Do we have any plan to enable VIP between NameNode and DataNodes, I mean
> > multiple NameNodes and we use a VIP stands before the NNs. Then DNs only
> > need to connect to the VIP. I don't know if it is valuable we have this
> impl
> > in our hdsf and if there has any issues?
> >
> > Any hints are valuable for me. :)
> >
> > regards
> > macf
> >
>
>
>
> --
> Harsh J
> www.harshj.com
>

Fb does not use the VIP approach, we tried that but quickly found out some
limitations, one main problem being that the failover server pair has to be
in the same subnet (for VIP to work). Instead we now use the AvatarNode
integrated with Zookeeper.

-dhruba

On Tue, Jan 25, 2011 at 6:12 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> This is a good approach, is used by many to mask the NameNode address
> for the DataNodes; and is also good to use while using BackupNode
> (Even Facebook does this).
>
> On Wed, Jan 26, 2011 at 7:38 AM, mac fang <[EMAIL PROTECTED]> wrote:
> > Hi, guys,
> >
> > Do we have any plan to enable VIP between NameNode and DataNodes, I mean
> > multiple NameNodes and we use a VIP stands before the NNs. Then DNs only
> > need to connect to the VIP. I don't know if it is valuable we have this
> impl
> > in our hdsf and if there has any issues?
> >
> > Any hints are valuable for me. :)
> >
> > regards
> > macf
> >
>
>
>
> --
> Harsh J
> www.harshj.com
>

--
Connect to me at http://www.facebook.com/dhruba

Thank you for correcting that :)

On Wed, Jan 26, 2011 at 11:17 AM, Dhruba Borthakur <[EMAIL PROTECTED]> wrote:
> Fb does not use the VIP approach, we tried that but quickly found out some
> limitations, one main problem being that the failover server pair has to be
> in the same subnet (for VIP to work). Instead we now use the AvatarNode
> integrated with Zookeeper.
>
> -dhruba
>
> On Tue, Jan 25, 2011 at 6:12 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> This is a good approach, is used by many to mask the NameNode address
>> for the DataNodes; and is also good to use while using BackupNode
>> (Even Facebook does this).
>>
>> On Wed, Jan 26, 2011 at 7:38 AM, mac fang <[EMAIL PROTECTED]> wrote:
>> > Hi, guys,
>> >
>> > Do we have any plan to enable VIP between NameNode and DataNodes, I mean
>> > multiple NameNodes and we use a VIP stands before the NNs. Then DNs only
>> > need to connect to the VIP. I don't know if it is valuable we have this
>> impl
>> > in our hdsf and if there has any issues?
>> >
>> > Any hints are valuable for me. :)
>> >
>> > regards
>> > macf
>> >
>>
>>
>>
>> --
>> Harsh J
>> www.harshj.com
>>
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>

--
Harsh J
www.harshj.com

Hi, dhruba,

Thanks for sharing. Seems we only need to let NameNode & Backup Node in the
same subnet. Why it is a main problem?

regards
macf

On Wed, Jan 26, 2011 at 4:14 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Thank you for correcting that :)
>
> On Wed, Jan 26, 2011 at 11:17 AM, Dhruba Borthakur <[EMAIL PROTECTED]>
> wrote:
> > Fb does not use the VIP approach, we tried that but quickly found out
> some
> > limitations, one main problem being that the failover server pair has to
> be
> > in the same subnet (for VIP to work). Instead we now use the AvatarNode
> > integrated with Zookeeper.
> >
> > -dhruba
> >
> > On Tue, Jan 25, 2011 at 6:12 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> >
> >> This is a good approach, is used by many to mask the NameNode address
> >> for the DataNodes; and is also good to use while using BackupNode
> >> (Even Facebook does this).
> >>
> >> On Wed, Jan 26, 2011 at 7:38 AM, mac fang <[EMAIL PROTECTED]> wrote:
> >> > Hi, guys,
> >> >
> >> > Do we have any plan to enable VIP between NameNode and DataNodes, I
> mean
> >> > multiple NameNodes and we use a VIP stands before the NNs. Then DNs
> only
> >> > need to connect to the VIP. I don't know if it is valuable we have
> this
> >> impl
> >> > in our hdsf and if there has any issues?
> >> >
> >> > Any hints are valuable for me. :)
> >> >
> >> > regards
> >> > macf
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >> www.harshj.com
> >>
> >
> >
> >
> > --
> > Connect to me at http://www.facebook.com/dhruba
> >
>
>
>
> --
> Harsh J
> www.harshj.com
>

If you want to protect urself against a rack-switch failure, then you would
want to keep ur primary and standby on machines in different racks.
Typically, machines in different racks have different subnets.

thanks
dhruba

On Wed, Jan 26, 2011 at 7:10 AM, mac fang <[EMAIL PROTECTED]> wrote:

> Hi, dhruba,
>
> Thanks for sharing. Seems we only need to let NameNode & Backup Node in the
> same subnet. Why it is a main problem?
>
> regards
> macf
>
> On Wed, Jan 26, 2011 at 4:14 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>
> > Thank you for correcting that :)
> >
> > On Wed, Jan 26, 2011 at 11:17 AM, Dhruba Borthakur <[EMAIL PROTECTED]>
> > wrote:
> > > Fb does not use the VIP approach, we tried that but quickly found out
> > some
> > > limitations, one main problem being that the failover server pair has
> to
> > be
> > > in the same subnet (for VIP to work). Instead we now use the AvatarNode
> > > integrated with Zookeeper.
> > >
> > > -dhruba
> > >
> > > On Tue, Jan 25, 2011 at 6:12 PM, Harsh J <[EMAIL PROTECTED]>
> wrote:
> > >
> > >> This is a good approach, is used by many to mask the NameNode address
> > >> for the DataNodes; and is also good to use while using BackupNode
> > >> (Even Facebook does this).
> > >>
> > >> On Wed, Jan 26, 2011 at 7:38 AM, mac fang <[EMAIL PROTECTED]>
> wrote:
> > >> > Hi, guys,
> > >> >
> > >> > Do we have any plan to enable VIP between NameNode and DataNodes, I
> > mean
> > >> > multiple NameNodes and we use a VIP stands before the NNs. Then DNs
> > only
> > >> > need to connect to the VIP. I don't know if it is valuable we have
> > this
> > >> impl
> > >> > in our hdsf and if there has any issues?
> > >> >
> > >> > Any hints are valuable for me. :)
> > >> >
> > >> > regards
> > >> > macf
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Harsh J
> > >> www.harshj.com
> > >>
> > >
> > >
> > >
> > > --
> > > Connect to me at http://www.facebook.com/dhruba
> > >
> >
> >
> >
> > --
> > Harsh J
> > www.harshj.com
> >
>

--
Connect to me at http://www.facebook.com/dhruba

FWIW, Y! and LI (and I'm sure others) have been using failover A records for years.  The only known problem/limitation is exactly what Dhruba states.

On Jan 29, 2011, at 11:19 PM, Dhruba Borthakur wrote:

> If you want to protect urself against a rack-switch failure, then you would
> want to keep ur primary and standby on machines in different racks.
> Typically, machines in different racks have different subnets.
>
> thanks
> dhruba
>
> On Wed, Jan 26, 2011 at 7:10 AM, mac fang <[EMAIL PROTECTED]> wrote:
>
>> Hi, dhruba,
>>
>> Thanks for sharing. Seems we only need to let NameNode & Backup Node in the
>> same subnet. Why it is a main problem?
>>
>> regards
>> macf
>>
>> On Wed, Jan 26, 2011 at 4:14 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>>> Thank you for correcting that :)
>>>
>>> On Wed, Jan 26, 2011 at 11:17 AM, Dhruba Borthakur <[EMAIL PROTECTED]>
>>> wrote:
>>>> Fb does not use the VIP approach, we tried that but quickly found out
>>> some
>>>> limitations, one main problem being that the failover server pair has
>> to
>>> be
>>>> in the same subnet (for VIP to work). Instead we now use the AvatarNode
>>>> integrated with Zookeeper.
>>>>
>>>> -dhruba
>>>>
>>>> On Tue, Jan 25, 2011 at 6:12 PM, Harsh J <[EMAIL PROTECTED]>
>> wrote:
>>>>
>>>>> This is a good approach, is used by many to mask the NameNode address
>>>>> for the DataNodes; and is also good to use while using BackupNode
>>>>> (Even Facebook does this).
>>>>>
>>>>> On Wed, Jan 26, 2011 at 7:38 AM, mac fang <[EMAIL PROTECTED]>
>> wrote:
>>>>>> Hi, guys,
>>>>>>
>>>>>> Do we have any plan to enable VIP between NameNode and DataNodes, I
>>> mean
>>>>>> multiple NameNodes and we use a VIP stands before the NNs. Then DNs
>>> only
>>>>>> need to connect to the VIP. I don't know if it is valuable we have
>>> this
>>>>> impl
>>>>>> in our hdsf and if there has any issues?
>>>>>>
>>>>>> Any hints are valuable for me. :)
>>>>>>
>>>>>> regards
>>>>>> macf
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>>> www.harshj.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Connect to me at http://www.facebook.com/dhruba
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>> www.harshj.com
>>>
>>
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba