Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Extra start-up overhead with hadoop-2.1.0-beta


Copy link to this message
-
Re: Extra start-up overhead with hadoop-2.1.0-beta
Hi Omkar,

  I have got these number by running a simple C program on the containers
that fetches the timestamp in microseconds and exits. The times mentioned
are low and high, they are not varying so drastically with in a version but
there are huge differences(like a second) between the two versions,
2.0.4-alpha and 2.1.0-beta as I mentioned.

  I am using a single node cluster, and all there is absolutely no other
load on the machine/node. My single node cluster is just used for my own
development work, and testing.

  I am not aware of what is resource localization, I am not doing anything
specially for that.

  Please let me know if you need any other info.

Thanks,
Kishore
On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <[EMAIL PROTECTED]> wrote:

> How are you making these measurements can you elaborate more? Is it on a
> best case basis or on an average or worst case? How many resources are you
> sending it for localization? were the sizes and number of these resources
> consistent across tests? Were these resources public/private/application
> specific? Apart from this is the other load on node manager same? is the
> load on hdfs same? did you see any network bottleneck?
>
> More information will help a lot.
>
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
> [EMAIL PROTECTED]> wrote:
>
>> Hi,
>>   Please share with me if you anyone has an answer or clues to my
>> question regarding the start up performance.
>>
>> Also, one more thing I have observed today is the time taken to run a
>> command on a container went up by more than a second in this latest version.
>>
>> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
>> I call startContainer() to the  point the command is started on the
>> container.
>>
>> where as
>>
>> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
>> came to the call back onContainerStarted() to the point the command is seen
>> started running on the container.
>>
>> Thanks,
>> Kishore
>>
>>
>> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>>
>>>   I have been using the hadoop-2.0.1-beta release candidate and observed
>>> that it is slower in running my simple application that runs on 2
>>> containers. I have tried to find out which parts of it is really having
>>> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
>>> found that.
>>>
>>> 1) From the point my Client has submitted the Application Master to RM,
>>> it is taking 2  seconds extra
>>> 2) From the point my container request are set up by Application Master,
>>> till the containers are allocated, it is taking 2 seconds extra
>>>
>>> Is this overhead expected with the changes that went into the new
>>> version? Or is there to improve it by changing something in configurations
>>> or so?
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>