Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Extra start-up overhead with hadoop-2.1.0-beta


Copy link to this message
-
Re: Extra start-up overhead with hadoop-2.1.0-beta
No Ravi, I am not running any MR job. Also, my configuration files are not
big.
On Wed, Aug 7, 2013 at 11:12 PM, Ravi Prakash <[EMAIL PROTECTED]> wrote:

> I believe https://issues.apache.org/jira/browse/MAPREDUCE-5399 causes
> performance degradation in cases where there are a lot of reducers. I can
> imagine it causing degradation if the configuration files are super big /
> some other weird cases.
>
>
>   ------------------------------
>  *From:* Krishna Kishore Bonagiri <[EMAIL PROTECTED]>
> *To:* [EMAIL PROTECTED]
> *Sent:* Wednesday, August 7, 2013 10:03 AM
> *Subject:* Re: Extra start-up overhead with hadoop-2.1.0-beta
>
> Hi Omkar,
>
>  Can you please see if you can answer my question with this info or if you
> need anything else from me?
>
>  Also, does resource localization improve or impact any performance?
>
> Thanks,
> Kishore
>
>
> On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <[EMAIL PROTECTED]>wrote:
>
> How are you making these measurements can you elaborate more? Is it on a
> best case basis or on an average or worst case? How many resources are you
> sending it for localization? were the sizes and number of these resources
> consistent across tests? Were these resources public/private/application
> specific? Apart from this is the other load on node manager same? is the
> load on hdfs same? did you see any network bottleneck?
>
> More information will help a lot.
>
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com/>
>
>
> On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
> [EMAIL PROTECTED]> wrote:
>
> Hi,
>   Please share with me if you anyone has an answer or clues to my question
> regarding the start up performance.
>
> Also, one more thing I have observed today is the time taken to run a
> command on a container went up by more than a second in this latest version.
>
> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
> I call startContainer() to the  point the command is started on the
> container.
>
> where as
>
> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
> came to the call back onContainerStarted() to the point the command is seen
> started running on the container.
>
> Thanks,
> Kishore
>
>
> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
> [EMAIL PROTECTED]> wrote:
>
> Hi,
>
>   I have been using the hadoop-2.0.1-beta release candidate and observed
> that it is slower in running my simple application that runs on 2
> containers. I have tried to find out which parts of it is really having
> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
> found that.
>
> 1) From the point my Client has submitted the Application Master to RM, it
> is taking 2  seconds extra
> 2) From the point my container request are set up by Application Master,
> till the containers are allocated, it is taking 2 seconds extra
>
> Is this overhead expected with the changes that went into the new version?
> Or is there to improve it by changing something in configurations or so?
>
> Thanks,
> Kishore
>
>
>
>
>
>
>