Matt Foley
2012-07-12, 21:26
Suresh Srinivas
2012-07-12, 21:45
Bhandarkar, Milind
2012-09-14, 18:44
-
update re: Hadoop-1.1.0-rc0Matt Foley 2012-07-12, 21:26
The following additional patches are committed to branch-1.1 and will be in
1.1.0:
- HADOOP-8365 - Add flag to disable durable sync
- HADOOP-8399 - Remove JDK5 dependency from Hadoop 1.0+ line
- HADOOP-8417 - HADOOP-6963 didn't update hadoop-core-pom-template.xml
- HADOOP-8445 - Token should not print the password in toString
- MAPREDUCE-2903 - Map Tasks graph is throwing XML Parse error when Job is
executed with 0 maps
- MAPREDUCE-2129 - Job may hang if
mapreduce.job.committer.setup.cleanup.needed=false and
mapreduce.map/reduce.failures.maxpercent>0
I've been trying to build and test the RC0, but am having build problems
with the 32-bit version. Should have it figured out soon.
The following patch is committed to branch-1 and I'll see if I can apply it
to branch-1.1 before my next build:
- HDFS-3617 - Port HDFS-96 to branch-1 (support blocks greater than 2GB)
The following patch is committed to branch-1 but had conflicts with
branch-1.1 and will not be in RC0:
- HDFS-3596 - Improve FSEditLog pre-allocation in branch-1
The following patches are deferred, as they do not have currently committed
patches:
- HADOOP-7823 - port HADOOP-4012 to branch-1 (splitting support for bzip2)
- MAPREDUCE-4049 - plugin for generic shuffle service
Thanks,
--Matt
On Sat, Jul 7, 2012 at 6:32 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
> Also, I have updated HADOOP-8417 against 1.1.0 and we need to include it.
> Otherwise, 1.1 will have the same issues for the downstream projects as
> 1.0.3
> had.
>
> Cos
>
> On Sat, Jul 07, 2012 at 05:52PM, Konstantin Boudnik wrote:
> > Matt,
> >
> > Thanks for the update.
> >
> > HADOOP-8399 would be beneficial for BigTop release and it is marked for
> 1.1.0
> > release. The patch is available for a while now and if someone can
> review I'd
> > go ahead and commit it today.
> >
> > I am working on the content of 0.3.1 BigTop release and will shortly
> post the
> > vote for it. Once Hadoop 1.1 rc is cut we'll start testing it with the
> rest of
> > the stack.
> >
> > Cos
> >
> > On Fri, Jul 06, 2012 at 02:24PM, Matt Foley wrote:
> > > Hi Cos,
> > > the query string didn't come thru on the link you sent, but the jira
> query
> > > I use is:
> > > project in (HADOOP,HDFS,MAPREDUCE) and (("Target Version/s" > '1.1.0'
> > > and (fixVersion != '1.1.0' or fixVersion is EMPTY)) or (fixVersion > > > '1.1.0' and "Target Version/s" is EMPTY)) and (status != Closed and
> status
> > > != Resolved) ORDER BY KEY
> > >
> > > You're correct that there are quite a few, currently 107, open jiras
> > > originally targeted for 1.1.0 that do not have committed fixes. Many
> of
> > > these are just the inherited backlog of previously identified work. I
> need
> > > to move them to "Target Version/s" = 1.1.1.
> > >
> > > Folks have requested that the following currently open jiras be
> included in
> > > 1.1.0:
> > >
> > > HADOOP-8417 - HADOOP-6963 didn't update hadoop-core-pom-template.xml
> > > HADOOP-8445 - Token should not print the password in toString
> > > HDFS-96 - HDFS does not support blocks greater than 2GB
> > > HDFS-3596 - Improve FSEditLog pre-allocation in branch-1
> > > MAPREDUCE-2903 - Map Tasks graph is throwing XML Parse error when Job
> is
> > > executed with 0 maps
> > > MAPREDUCE-2129 - Job may hang if
> > > mapreduce.job.committer.setup.cleanup.needed=false and
> > > mapreduce.map/reduce.failures.maxpercent>0
> > > ---
> > > MAPREDUCE-4049 - plugin for generic shuffle service
> > > HADOOP-7823 - port HADOOP-4012 to branch-1 (splitting support for
> bzip2)
> > >
> > > The first six are simple patches that I am comfortable including.
> > > The last two are complex patches that have not yet been committed.
> > > I am planning to defer those two to 1.1.1.
> > >
> > > Beyond that, I'm going to cut 1.1.0-rc0 from the current state of
> > > branch-1.1.
> > > I'm planning to do that this weekend. This is obviously delayed from
> the
> > > previous plan, for which I apologize.
> > >
> > > Comments welcome.
1.1.0:
- HADOOP-8365 - Add flag to disable durable sync
- HADOOP-8399 - Remove JDK5 dependency from Hadoop 1.0+ line
- HADOOP-8417 - HADOOP-6963 didn't update hadoop-core-pom-template.xml
- HADOOP-8445 - Token should not print the password in toString
- MAPREDUCE-2903 - Map Tasks graph is throwing XML Parse error when Job is
executed with 0 maps
- MAPREDUCE-2129 - Job may hang if
mapreduce.job.committer.setup.cleanup.needed=false and
mapreduce.map/reduce.failures.maxpercent>0
I've been trying to build and test the RC0, but am having build problems
with the 32-bit version. Should have it figured out soon.
The following patch is committed to branch-1 and I'll see if I can apply it
to branch-1.1 before my next build:
- HDFS-3617 - Port HDFS-96 to branch-1 (support blocks greater than 2GB)
The following patch is committed to branch-1 but had conflicts with
branch-1.1 and will not be in RC0:
- HDFS-3596 - Improve FSEditLog pre-allocation in branch-1
The following patches are deferred, as they do not have currently committed
patches:
- HADOOP-7823 - port HADOOP-4012 to branch-1 (splitting support for bzip2)
- MAPREDUCE-4049 - plugin for generic shuffle service
Thanks,
--Matt
On Sat, Jul 7, 2012 at 6:32 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
> Also, I have updated HADOOP-8417 against 1.1.0 and we need to include it.
> Otherwise, 1.1 will have the same issues for the downstream projects as
> 1.0.3
> had.
>
> Cos
>
> On Sat, Jul 07, 2012 at 05:52PM, Konstantin Boudnik wrote:
> > Matt,
> >
> > Thanks for the update.
> >
> > HADOOP-8399 would be beneficial for BigTop release and it is marked for
> 1.1.0
> > release. The patch is available for a while now and if someone can
> review I'd
> > go ahead and commit it today.
> >
> > I am working on the content of 0.3.1 BigTop release and will shortly
> post the
> > vote for it. Once Hadoop 1.1 rc is cut we'll start testing it with the
> rest of
> > the stack.
> >
> > Cos
> >
> > On Fri, Jul 06, 2012 at 02:24PM, Matt Foley wrote:
> > > Hi Cos,
> > > the query string didn't come thru on the link you sent, but the jira
> query
> > > I use is:
> > > project in (HADOOP,HDFS,MAPREDUCE) and (("Target Version/s" > '1.1.0'
> > > and (fixVersion != '1.1.0' or fixVersion is EMPTY)) or (fixVersion > > > '1.1.0' and "Target Version/s" is EMPTY)) and (status != Closed and
> status
> > > != Resolved) ORDER BY KEY
> > >
> > > You're correct that there are quite a few, currently 107, open jiras
> > > originally targeted for 1.1.0 that do not have committed fixes. Many
> of
> > > these are just the inherited backlog of previously identified work. I
> need
> > > to move them to "Target Version/s" = 1.1.1.
> > >
> > > Folks have requested that the following currently open jiras be
> included in
> > > 1.1.0:
> > >
> > > HADOOP-8417 - HADOOP-6963 didn't update hadoop-core-pom-template.xml
> > > HADOOP-8445 - Token should not print the password in toString
> > > HDFS-96 - HDFS does not support blocks greater than 2GB
> > > HDFS-3596 - Improve FSEditLog pre-allocation in branch-1
> > > MAPREDUCE-2903 - Map Tasks graph is throwing XML Parse error when Job
> is
> > > executed with 0 maps
> > > MAPREDUCE-2129 - Job may hang if
> > > mapreduce.job.committer.setup.cleanup.needed=false and
> > > mapreduce.map/reduce.failures.maxpercent>0
> > > ---
> > > MAPREDUCE-4049 - plugin for generic shuffle service
> > > HADOOP-7823 - port HADOOP-4012 to branch-1 (splitting support for
> bzip2)
> > >
> > > The first six are simple patches that I am comfortable including.
> > > The last two are complex patches that have not yet been committed.
> > > I am planning to defer those two to 1.1.1.
> > >
> > > Beyond that, I'm going to cut 1.1.0-rc0 from the current state of
> > > branch-1.1.
> > > I'm planning to do that this weekend. This is obviously delayed from
> the
> > > previous plan, for which I apologize.
> > >
> > > Comments welcome.
-
Re: update re: Hadoop-1.1.0-rc0Suresh Srinivas 2012-07-12, 21:45
Matt,
Meanwhile, I am planning to promote the following changes:
HDFS-2465 - Add HDFS support for fadvise readahead and
drop-behind<https://issues.apache.org/jira/browse/HDFS-2465>
HADOOP-Support fadvise and sync_data_range in NativeIO, add ReadaheadPool
class <https://issues.apache.org/jira/browse/HADOOP-7753>
These changes should be low risk - These i/o improvements are turned off by
default in configuration. Brandon Li has done quite a bit of testing
(TestDFSIO, teragen, terasort, teravalidate etc.) on a small cluster (3
nodes) in addition to unit tests.
Regards,
Suresh
On Thu, Jul 12, 2012 at 2:26 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
> The following additional patches are committed to branch-1.1 and will be in
> 1.1.0:
> - HADOOP-8365 - Add flag to disable durable sync
> - HADOOP-8399 - Remove JDK5 dependency from Hadoop 1.0+ line
> - HADOOP-8417 - HADOOP-6963 didn't update hadoop-core-pom-template.xml
> - HADOOP-8445 - Token should not print the password in toString
> - MAPREDUCE-2903 - Map Tasks graph is throwing XML Parse error when Job is
> executed with 0 maps
> - MAPREDUCE-2129 - Job may hang if
> mapreduce.job.committer.setup.cleanup.needed=false and
> mapreduce.map/reduce.failures.maxpercent>0
>
> I've been trying to build and test the RC0, but am having build problems
> with the 32-bit version. Should have it figured out soon.
>
> The following patch is committed to branch-1 and I'll see if I can apply it
> to branch-1.1 before my next build:
> - HDFS-3617 - Port HDFS-96 to branch-1 (support blocks greater than 2GB)
>
> The following patch is committed to branch-1 but had conflicts with
> branch-1.1 and will not be in RC0:
> - HDFS-3596 - Improve FSEditLog pre-allocation in branch-1
>
> The following patches are deferred, as they do not have currently committed
> patches:
> - HADOOP-7823 - port HADOOP-4012 to branch-1 (splitting support for bzip2)
> - MAPREDUCE-4049 - plugin for generic shuffle service
>
> Thanks,
> --Matt
>
> On Sat, Jul 7, 2012 at 6:32 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
>
> > Also, I have updated HADOOP-8417 against 1.1.0 and we need to include it.
> > Otherwise, 1.1 will have the same issues for the downstream projects as
> > 1.0.3
> > had.
> >
> > Cos
> >
> > On Sat, Jul 07, 2012 at 05:52PM, Konstantin Boudnik wrote:
> > > Matt,
> > >
> > > Thanks for the update.
> > >
> > > HADOOP-8399 would be beneficial for BigTop release and it is marked for
> > 1.1.0
> > > release. The patch is available for a while now and if someone can
> > review I'd
> > > go ahead and commit it today.
> > >
> > > I am working on the content of 0.3.1 BigTop release and will shortly
> > post the
> > > vote for it. Once Hadoop 1.1 rc is cut we'll start testing it with the
> > rest of
> > > the stack.
> > >
> > > Cos
> > >
> > > On Fri, Jul 06, 2012 at 02:24PM, Matt Foley wrote:
> > > > Hi Cos,
> > > > the query string didn't come thru on the link you sent, but the jira
> > query
> > > > I use is:
> > > > project in (HADOOP,HDFS,MAPREDUCE) and (("Target Version/s" > > '1.1.0'
> > > > and (fixVersion != '1.1.0' or fixVersion is EMPTY)) or (fixVersion > > > > '1.1.0' and "Target Version/s" is EMPTY)) and (status != Closed and
> > status
> > > > != Resolved) ORDER BY KEY
> > > >
> > > > You're correct that there are quite a few, currently 107, open jiras
> > > > originally targeted for 1.1.0 that do not have committed fixes. Many
> > of
> > > > these are just the inherited backlog of previously identified work.
> I
> > need
> > > > to move them to "Target Version/s" = 1.1.1.
> > > >
> > > > Folks have requested that the following currently open jiras be
> > included in
> > > > 1.1.0:
> > > >
> > > > HADOOP-8417 - HADOOP-6963 didn't update hadoop-core-pom-template.xml
> > > > HADOOP-8445 - Token should not print the password in toString
> > > > HDFS-96 - HDFS does not support blocks greater than 2GB
> > > > HDFS-3596 - Improve FSEditLog pre-allocation in branch-1
> > > > MAPREDUCE-2903 - Map Tasks graph is throwing XML Parse error when Job
http://hortonworks.com/download/
Meanwhile, I am planning to promote the following changes:
HDFS-2465 - Add HDFS support for fadvise readahead and
drop-behind<https://issues.apache.org/jira/browse/HDFS-2465>
HADOOP-Support fadvise and sync_data_range in NativeIO, add ReadaheadPool
class <https://issues.apache.org/jira/browse/HADOOP-7753>
These changes should be low risk - These i/o improvements are turned off by
default in configuration. Brandon Li has done quite a bit of testing
(TestDFSIO, teragen, terasort, teravalidate etc.) on a small cluster (3
nodes) in addition to unit tests.
Regards,
Suresh
On Thu, Jul 12, 2012 at 2:26 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
> The following additional patches are committed to branch-1.1 and will be in
> 1.1.0:
> - HADOOP-8365 - Add flag to disable durable sync
> - HADOOP-8399 - Remove JDK5 dependency from Hadoop 1.0+ line
> - HADOOP-8417 - HADOOP-6963 didn't update hadoop-core-pom-template.xml
> - HADOOP-8445 - Token should not print the password in toString
> - MAPREDUCE-2903 - Map Tasks graph is throwing XML Parse error when Job is
> executed with 0 maps
> - MAPREDUCE-2129 - Job may hang if
> mapreduce.job.committer.setup.cleanup.needed=false and
> mapreduce.map/reduce.failures.maxpercent>0
>
> I've been trying to build and test the RC0, but am having build problems
> with the 32-bit version. Should have it figured out soon.
>
> The following patch is committed to branch-1 and I'll see if I can apply it
> to branch-1.1 before my next build:
> - HDFS-3617 - Port HDFS-96 to branch-1 (support blocks greater than 2GB)
>
> The following patch is committed to branch-1 but had conflicts with
> branch-1.1 and will not be in RC0:
> - HDFS-3596 - Improve FSEditLog pre-allocation in branch-1
>
> The following patches are deferred, as they do not have currently committed
> patches:
> - HADOOP-7823 - port HADOOP-4012 to branch-1 (splitting support for bzip2)
> - MAPREDUCE-4049 - plugin for generic shuffle service
>
> Thanks,
> --Matt
>
> On Sat, Jul 7, 2012 at 6:32 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
>
> > Also, I have updated HADOOP-8417 against 1.1.0 and we need to include it.
> > Otherwise, 1.1 will have the same issues for the downstream projects as
> > 1.0.3
> > had.
> >
> > Cos
> >
> > On Sat, Jul 07, 2012 at 05:52PM, Konstantin Boudnik wrote:
> > > Matt,
> > >
> > > Thanks for the update.
> > >
> > > HADOOP-8399 would be beneficial for BigTop release and it is marked for
> > 1.1.0
> > > release. The patch is available for a while now and if someone can
> > review I'd
> > > go ahead and commit it today.
> > >
> > > I am working on the content of 0.3.1 BigTop release and will shortly
> > post the
> > > vote for it. Once Hadoop 1.1 rc is cut we'll start testing it with the
> > rest of
> > > the stack.
> > >
> > > Cos
> > >
> > > On Fri, Jul 06, 2012 at 02:24PM, Matt Foley wrote:
> > > > Hi Cos,
> > > > the query string didn't come thru on the link you sent, but the jira
> > query
> > > > I use is:
> > > > project in (HADOOP,HDFS,MAPREDUCE) and (("Target Version/s" > > '1.1.0'
> > > > and (fixVersion != '1.1.0' or fixVersion is EMPTY)) or (fixVersion > > > > '1.1.0' and "Target Version/s" is EMPTY)) and (status != Closed and
> > status
> > > > != Resolved) ORDER BY KEY
> > > >
> > > > You're correct that there are quite a few, currently 107, open jiras
> > > > originally targeted for 1.1.0 that do not have committed fixes. Many
> > of
> > > > these are just the inherited backlog of previously identified work.
> I
> > need
> > > > to move them to "Target Version/s" = 1.1.1.
> > > >
> > > > Folks have requested that the following currently open jiras be
> > included in
> > > > 1.1.0:
> > > >
> > > > HADOOP-8417 - HADOOP-6963 didn't update hadoop-core-pom-template.xml
> > > > HADOOP-8445 - Token should not print the password in toString
> > > > HDFS-96 - HDFS does not support blocks greater than 2GB
> > > > HDFS-3596 - Improve FSEditLog pre-allocation in branch-1
> > > > MAPREDUCE-2903 - Map Tasks graph is throwing XML Parse error when Job
http://hortonworks.com/download/
-
Re: update re: Hadoop-1.1.0-rc0Bhandarkar, Milind 2012-09-14, 18:44
>- MAPREDUCE-4049 - plugin for generic shuffle service
Committers,
What else is needed on MR-4049 to commit ?
- milind
>
