Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Re-swizzle 2.3


Copy link to this message
-
Re: Re-swizzle 2.3
I think we should revert YARN-1490 from Hadoop 2.3 branch.  I think it was
causing some strange behavior in the Oozie unit tests:

Basically, we use a single MiniMRCluster and MiniDFSCluster across all unit
tests in a module.  With YARN-1490 we saw that, regardless of test order,
the last few tests would timeout waiting for an MR job to finish; on slower
machines, the entire test suite would timeout.  Through some digging, I
found that we were getting a ton of "Connection refused" Exceptions on
LeaseRenewer talking to the NN and a few on the AM talking to the RM.

After a bunch of investigation, I found that the problem went away once
YARN-1490 was removed.  Though I couldn't figure out the exact problem.
 Even though this occurred in unit tests, it does make me concerned that it
could indicate some bigger issue in a long-running real cluster (where
everything isn't running on the same machine) that we haven't seen yet.

On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <[EMAIL PROTECTED]> wrote:

> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly. Will
> pull it out of branch-2.3 if anyone objects.
>
>
> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <[EMAIL PROTECTED]
> >wrote:
>
> > Merged HADOOP-10273 to branch-2.3 as r1565456.
> >
> >
> > On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <[EMAIL PROTECTED]
> > >wrote:
> >
> > > IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> > >
> > > I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> > >
> > >
> > > On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <[EMAIL PROTECTED]
> > >wrote:
> > >
> > >> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very odd
> > >> ways
> > >> (to the point it seems un-deterministic).
> > >>
> > >> I'd say eiher YARN-1577 is fixed or we revert
> > >> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> reverts)
> > >> from Hadoop 2.3 branch before doing the release.
> > >>
> > >>
> > >> I've verified that after reverting those JIRAs things work fine with
> > >> unmanaged AMs.
> > >>
> > >> Thanks.
> > >>
> > >>
> > >>
> > >>
> > >> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <[EMAIL PROTECTED]>
> > >> wrote:
> > >>
> > >> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
> > >> >
> > >> > Jian is away and I don't see YARN-1577 & YARN-1206 making much
> > progress
> > >> > till he is back; so I'm inclined to push both to 2.4 too. Any
> > >> objections?
> > >> >
> > >> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> > >> >
> > >> > Overall, I'll try get this out in next couple of days if we can
> clear
> > >> the
> > >> > list.
> > >> >
> > >> > thanks,
> > >> > Arun
> > >> >
> > >> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <[EMAIL PROTECTED]>
> > wrote:
> > >> >
> > >> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are
> > now
> > >> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> > >> > >
> > >> > > Daryn (thanks!) has both the non-YARN covered. Vinod is helping
> out
> > >> with
> > >> > the YARN ones.
> > >> > >
> > >> > > thanks,
> > >> > > Arun
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >> > --
> > >> > Arun C. Murthy
> > >> > Hortonworks Inc.
> > >> > http://hortonworks.com/
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > CONFIDENTIALITY NOTICE
> > >> > NOTICE: This message is intended for the use of the individual or
> > >> entity to
> > >> > which it is addressed and may contain information that is
> > confidential,
> > >> > privileged and exempt from disclosure under applicable law. If the
> > >> reader
> > >> > of this message is not the intended recipient, you are hereby
> notified
> > >> that
> > >> > any printing, copying, dissemination, distribution, disclosure or
> > >> > forwarding of this communication is strictly prohibited. If you have
> > >> > received this communication in error, please contact the sender
> > >> immediately
> > >> > and delete it from your system. Thank You.
> > >