-Re: Extra start-up overhead with hadoop-2.1.0-beta
Krishna Kishore Bonagiri 2013-08-02, 13:52
I have got these number by running a simple C program on the containers
that fetches the timestamp in microseconds and exits. The times mentioned
are low and high, they are not varying so drastically with in a version but
there are huge differences(like a second) between the two versions,
2.0.4-alpha and 2.1.0-beta as I mentioned.
I am using a single node cluster, and all there is absolutely no other
load on the machine/node. My single node cluster is just used for my own
development work, and testing.
I am not aware of what is resource localization, I am not doing anything
specially for that.
Please let me know if you need any other info.
On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <[EMAIL PROTECTED]> wrote:
> How are you making these measurements can you elaborate more? Is it on a
> best case basis or on an average or worst case? How many resources are you
> sending it for localization? were the sizes and number of these resources
> consistent across tests? Were these resources public/private/application
> specific? Apart from this is the other load on node manager same? is the
> load on hdfs same? did you see any network bottleneck?
> More information will help a lot.
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
> On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
> [EMAIL PROTECTED]> wrote:
>> Please share with me if you anyone has an answer or clues to my
>> question regarding the start up performance.
>> Also, one more thing I have observed today is the time taken to run a
>> command on a container went up by more than a second in this latest version.
>> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
>> I call startContainer() to the point the command is started on the
>> where as
>> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
>> came to the call back onContainerStarted() to the point the command is seen
>> started running on the container.
>> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
>> [EMAIL PROTECTED]> wrote:
>>> I have been using the hadoop-2.0.1-beta release candidate and observed
>>> that it is slower in running my simple application that runs on 2
>>> containers. I have tried to find out which parts of it is really having
>>> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
>>> found that.
>>> 1) From the point my Client has submitted the Application Master to RM,
>>> it is taking 2 seconds extra
>>> 2) From the point my container request are set up by Application Master,
>>> till the containers are allocated, it is taking 2 seconds extra
>>> Is this overhead expected with the changes that went into the new
>>> version? Or is there to improve it by changing something in configurations
>>> or so?