Hi,

I know that DataNode and TaskTracker must restart to change topology.
Is there the method to execute the topology change without restart of
DataNode and TaskTracker?
In other words, can I change the topology by a command?

Thanks in advance!
Shinichi

On 8 October 2012 14:23, Shinichi Yamashita <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I know that DataNode and TaskTracker must restart to change topology.
>

no, it's the Namenode and JobTracker that needs to be restarted; they are
the bits that care where the boxes are.
> Is there the method to execute the topology change without restart of
> DataNode and TaskTracker?
> In other words, can I change the topology by a command?
>

No -the map gets cached, and when the services start up they do quick check
to see if the topology is flat or spread across switches -and use that flag
to make various placement decisions.

steve

On 9 October 2012 16:51, Shinichi Yamashita <[EMAIL PROTECTED]> wrote:

> Hi Steve,
>
> Thank you for your reply.
>
>
> > no, it's the Namenode and JobTracker that needs to be restarted;
> > they are the bits that care where the boxes are.
>
> I confirmed it in my cluster, and I understood it as follows.
> First, the resolved node information is recorded in ConcurrentHashMap.
> Next same node toplogy decision, HashMap data is given priority to over a
> script.
> The HashMap data become invalid by restart of NameNode and JobTracker.
>

That's right. There's been lots of discussion on adding live topology
resetting.

There's also a large unfinished JIRA in my name to do better topology work
in Hadoop 2,
https://issues.apache.org/jira/browse/HADOOP-8231

Including the ability to do local checks of a topology script
https://issues.apache.org/jira/browse/HADOOP-8232

Supporting topology dump and reset from any of the manager nodes would come
after.

-steve

On Tue, Oct 9, 2012 at 12:17 PM, Steve Loughran <[EMAIL PROTECTED]>wrote:

>
>
> On 9 October 2012 16:51, Shinichi Yamashita <[EMAIL PROTECTED]> wrote:
>
>> Hi Steve,
>>
>> Thank you for your reply.
>>
>>
>> > no, it's the Namenode and JobTracker that needs to be restarted;
>> > they are the bits that care where the boxes are.
>>
>> I confirmed it in my cluster, and I understood it as follows.
>> First, the resolved node information is recorded in ConcurrentHashMap.
>> Next same node toplogy decision, HashMap data is given priority to over a
>> script.
>> The HashMap data become invalid by restart of NameNode and JobTracker.
>>
>
> That's right. There's been lots of discussion on adding live topology
> resetting.
>

For what it's worth, there is at least one Hadoop derived distribution
which has live topology resetting already.

I can follow up offline with anyone who wants more information.

Hi Steve,

Thank you for sharing information of JIRA with me.
Topology easy setting and confirmation will be necessary.
I use JIRA for an idea about toplogy setting and confirmation.

Regards,
Shinichi

2012/10/10 Ted Dunning <[EMAIL PROTECTED]>

>
>
> On Tue, Oct 9, 2012 at 12:17 PM, Steve Loughran <[EMAIL PROTECTED]>wrote:
>
>>
>>
>> On 9 October 2012 16:51, Shinichi Yamashita <[EMAIL PROTECTED]> wrote:
>>
>>> Hi Steve,
>>>
>>> Thank you for your reply.
>>>
>>>
>>> > no, it's the Namenode and JobTracker that needs to be restarted;
>>> > they are the bits that care where the boxes are.
>>>
>>> I confirmed it in my cluster, and I understood it as follows.
>>> First, the resolved node information is recorded in ConcurrentHashMap.
>>> Next same node toplogy decision, HashMap data is given priority to over
>>> a script.
>>> The HashMap data become invalid by restart of NameNode and JobTracker.
>>>
>>
>> That's right. There's been lots of discussion on adding live topology
>> resetting.
>>
>
> For what it's worth, there is at least one Hadoop derived distribution
> which has live topology resetting already.
>
> I can follow up offline with anyone who wants more information.
>
>