-
Re: .23 compile timesAlejandro Abdelnur 2012-02-17, 00:15
A 'mvn package -DskipTests -Pdist' (developer) with no changes takes
3.35mins
The biggest offenders are:
[INFO] Apache Hadoop Common .............................. SUCCESS [43.299s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [54.776s]
[INFO] hadoop-yarn-api ................................... SUCCESS [15.466s]
[INFO] hadoop-mapreduce .................................. SUCCESS [16.154s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [9.657s]
Everybody else is below 5secs
A 'mvn test -DskipTests' with no changes 1.05mins (this should go down
making the some codegen to be smart instead just compiling again)
The biggest offenders are:
[INFO] Apache Hadoop Common .............................. SUCCESS [9.121s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [20.887s]
[INFO] hadoop-yarn-api ................................... SUCCESS [6.664s]
A few 2sec modules and the rest below 1sec
Thanks.
Alejandro
On Thu, Feb 16, 2012 at 2:13 PM, Daryn Sharp <[EMAIL PROTECTED]> wrote:
> Wow, nice! How long is a rebuild with no changes?
>
> I'd be delighted if there was a dev profile that just did the bare minimum
> to produce a working env.
>
> Daryn
>
>
> On Feb 16, 2012, at 3:25 PM, Alejandro Abdelnur wrote:
>
> > I've just done some tweaks in the POMs and I'm cutting the 8.30min full
> > dist build down to 3.45min. I'm removing all javadoc stuff from the dist
> > profile.
> >
> > I think we should introduce a developer profile that does this.
> >
> > Also, the 'mvn install -DskipTests', to get the JARs in your m2 cache is
> > taking 2mins. But we are building source JARs, the developer profile
> could
> > skip the generation of the source JARs, this would cut several seconds.
> >
> > Finally, note that to just build all JARs 'mvn clean test -DskipTests' it
> > is taking 1.55mins.
> >
> > If folk agree I'll open a JIRA to optmize the build based on these
> findings.
> >
> > Thxs.
> >
> > Alejandro
> >
> > On Wed, Feb 15, 2012 at 1:02 PM, Daryn Sharp <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hi Sriram,
> >>
> >> You may want to try adding your source tree to Spotlight's privacy list,
> >> otherwise it's busy trying to index files as fast as the build is
> writing
> >> them. If you are (un)fortunate enough to be running anti-virus sw,
> then it
> >> too is scurrying along behind your build checking every file. That's
> >> enough to turn a normally silent MBP into a hair dryer.
> >>
> >> If you don't want to omit the entire project, because perhaps you want
> to
> >> use mdfind, then you can rename dirs to end with ".noindex". The
> >> trick/hack is you can symlink the original dir name to the renamed
> >> dir.noindex.
> >>
> >> But yes, I think the build is way to slow to be productive when used as
> >> intended. The fact that people have to use custom tricks to build &
> test
> >> in a "reasonable" amount of time is a symptom of a problem...
> >>
> >> Daryn
> >>
> >>
> >> On Feb 15, 2012, at 12:47 PM, Todd Lipcon wrote:
> >>
> >>> Hi Sriram,
> >>>
> >>> I also do -Dmaven.skip.javadoc=true usually and that cuts out a couple
> >>> minutes. Unless you need a tarball, you can also drop -Ptar.
> >>>
> >>> When I'm just making quick changes confined to the java code, I
> >>> usually just use "mvn -Dskiptests install" from within the hdfs
> >>> project, then manually cp the resulting
> >>> target/hadoop-hdfs-0.24.0-SNAPSHOT.jar into my install
> >>> dir/share/hadoop/hdfs. Much faster than the full rebuild.
> >>>
> >>> -Todd
> >>>
> >>> On Wed, Feb 15, 2012 at 8:52 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> >>>> Sriram,
> >>>>
> >>>> My package command didn't take as long with the same command -- where
> >>>> do you see it getting stuck at most of the times? Which particular
> >>>> target (right term?) seems to take the highest time to complete?
> >>>>
> >>>> On Wed, Feb 15, 2012 at 12:31 AM, Sriram Rao <[EMAIL PROTECTED]>
> >> wrote:
> >>>>> Folks,
> >>>>>
> >>>>> The build takes forever:
3.35mins
The biggest offenders are:
[INFO] Apache Hadoop Common .............................. SUCCESS [43.299s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [54.776s]
[INFO] hadoop-yarn-api ................................... SUCCESS [15.466s]
[INFO] hadoop-mapreduce .................................. SUCCESS [16.154s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [9.657s]
Everybody else is below 5secs
A 'mvn test -DskipTests' with no changes 1.05mins (this should go down
making the some codegen to be smart instead just compiling again)
The biggest offenders are:
[INFO] Apache Hadoop Common .............................. SUCCESS [9.121s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [20.887s]
[INFO] hadoop-yarn-api ................................... SUCCESS [6.664s]
A few 2sec modules and the rest below 1sec
Thanks.
Alejandro
On Thu, Feb 16, 2012 at 2:13 PM, Daryn Sharp <[EMAIL PROTECTED]> wrote:
> Wow, nice! How long is a rebuild with no changes?
>
> I'd be delighted if there was a dev profile that just did the bare minimum
> to produce a working env.
>
> Daryn
>
>
> On Feb 16, 2012, at 3:25 PM, Alejandro Abdelnur wrote:
>
> > I've just done some tweaks in the POMs and I'm cutting the 8.30min full
> > dist build down to 3.45min. I'm removing all javadoc stuff from the dist
> > profile.
> >
> > I think we should introduce a developer profile that does this.
> >
> > Also, the 'mvn install -DskipTests', to get the JARs in your m2 cache is
> > taking 2mins. But we are building source JARs, the developer profile
> could
> > skip the generation of the source JARs, this would cut several seconds.
> >
> > Finally, note that to just build all JARs 'mvn clean test -DskipTests' it
> > is taking 1.55mins.
> >
> > If folk agree I'll open a JIRA to optmize the build based on these
> findings.
> >
> > Thxs.
> >
> > Alejandro
> >
> > On Wed, Feb 15, 2012 at 1:02 PM, Daryn Sharp <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hi Sriram,
> >>
> >> You may want to try adding your source tree to Spotlight's privacy list,
> >> otherwise it's busy trying to index files as fast as the build is
> writing
> >> them. If you are (un)fortunate enough to be running anti-virus sw,
> then it
> >> too is scurrying along behind your build checking every file. That's
> >> enough to turn a normally silent MBP into a hair dryer.
> >>
> >> If you don't want to omit the entire project, because perhaps you want
> to
> >> use mdfind, then you can rename dirs to end with ".noindex". The
> >> trick/hack is you can symlink the original dir name to the renamed
> >> dir.noindex.
> >>
> >> But yes, I think the build is way to slow to be productive when used as
> >> intended. The fact that people have to use custom tricks to build &
> test
> >> in a "reasonable" amount of time is a symptom of a problem...
> >>
> >> Daryn
> >>
> >>
> >> On Feb 15, 2012, at 12:47 PM, Todd Lipcon wrote:
> >>
> >>> Hi Sriram,
> >>>
> >>> I also do -Dmaven.skip.javadoc=true usually and that cuts out a couple
> >>> minutes. Unless you need a tarball, you can also drop -Ptar.
> >>>
> >>> When I'm just making quick changes confined to the java code, I
> >>> usually just use "mvn -Dskiptests install" from within the hdfs
> >>> project, then manually cp the resulting
> >>> target/hadoop-hdfs-0.24.0-SNAPSHOT.jar into my install
> >>> dir/share/hadoop/hdfs. Much faster than the full rebuild.
> >>>
> >>> -Todd
> >>>
> >>> On Wed, Feb 15, 2012 at 8:52 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> >>>> Sriram,
> >>>>
> >>>> My package command didn't take as long with the same command -- where
> >>>> do you see it getting stuck at most of the times? Which particular
> >>>> target (right term?) seems to take the highest time to complete?
> >>>>
> >>>> On Wed, Feb 15, 2012 at 12:31 AM, Sriram Rao <[EMAIL PROTECTED]>
> >> wrote:
> >>>>> Folks,
> >>>>>
> >>>>> The build takes forever:
