Folks,

The build takes forever:

 mvn package -Pdist -DskipTests -Dtar

....

[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 6:25.399s
[INFO] Finished at: Tue Feb 14 10:54:18 PST 2012
[INFO] Final Memory: 56M/123M
[INFO]
------------------------------------------------------------------------

This is on a mac with 8GB RAM.  (The above case involved 0-lines of code
change; just making sure I had everything built!).

Is there a faster way to get the thing to build?

Sriram

Sriram,

My package command didn't take as long with the same command -- where
do you see it getting stuck at most of the times? Which particular
target (right term?) seems to take the highest time to complete?

On Wed, Feb 15, 2012 at 12:31 AM, Sriram Rao <[EMAIL PROTECTED]> wrote:
> Folks,
>
> The build takes forever:
>
>  mvn package -Pdist -DskipTests -Dtar
>
> ....
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 6:25.399s
> [INFO] Finished at: Tue Feb 14 10:54:18 PST 2012
> [INFO] Final Memory: 56M/123M
> [INFO]
> ------------------------------------------------------------------------
>
> This is on a mac with 8GB RAM.  (The above case involved 0-lines of code
> change; just making sure I had everything built!).
>
> Is there a faster way to get the thing to build?
>
> Sriram

--
Harsh J
Customer Ops. Engineer
Cloudera | http://tiny.cloudera.com/about

Hi Sriram,

I also do -Dmaven.skip.javadoc=true usually and that cuts out a couple
minutes. Unless you need a tarball, you can also drop -Ptar.

When I'm just making quick changes confined to the java code, I
usually just use "mvn -Dskiptests install" from within the hdfs
project, then manually cp the resulting
target/hadoop-hdfs-0.24.0-SNAPSHOT.jar into my install
dir/share/hadoop/hdfs. Much faster than the full rebuild.

-Todd

On Wed, Feb 15, 2012 at 8:52 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> Sriram,
>
> My package command didn't take as long with the same command -- where
> do you see it getting stuck at most of the times? Which particular
> target (right term?) seems to take the highest time to complete?
>
> On Wed, Feb 15, 2012 at 12:31 AM, Sriram Rao <[EMAIL PROTECTED]> wrote:
>> Folks,
>>
>> The build takes forever:
>>
>>  mvn package -Pdist -DskipTests -Dtar
>>
>> ....
>>
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD SUCCESS
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 6:25.399s
>> [INFO] Finished at: Tue Feb 14 10:54:18 PST 2012
>> [INFO] Final Memory: 56M/123M
>> [INFO]
>> ------------------------------------------------------------------------
>>
>> This is on a mac with 8GB RAM.  (The above case involved 0-lines of code
>> change; just making sure I had everything built!).
>>
>> Is there a faster way to get the thing to build?
>>
>> Sriram
>
>
>
> --
> Harsh J
> Customer Ops. Engineer
> Cloudera | http://tiny.cloudera.com/about

--
Todd Lipcon
Software Engineer, Cloudera

Hi Sriram,

You may want to try adding your source tree to Spotlight's privacy list, otherwise it's busy trying to index files as fast as the build is writing them.  If you are (un)fortunate enough to be running anti-virus sw, then it too is scurrying along behind your build checking every file.  That's enough to turn a normally silent MBP into a hair dryer.

If you don't want to omit the entire project, because perhaps you want to use mdfind, then you can rename dirs to end with ".noindex".  The trick/hack is you can symlink the original dir name to the renamed dir.noindex.

But yes, I think the build is way to slow to be productive when used as intended.  The fact that people have to use custom tricks to build & test in a "reasonable" amount of time is a symptom of a problem...

Daryn
On Feb 15, 2012, at 12:47 PM, Todd Lipcon wrote:

> Hi Sriram,
>
> I also do -Dmaven.skip.javadoc=true usually and that cuts out a couple
> minutes. Unless you need a tarball, you can also drop -Ptar.
>
> When I'm just making quick changes confined to the java code, I
> usually just use "mvn -Dskiptests install" from within the hdfs
> project, then manually cp the resulting
> target/hadoop-hdfs-0.24.0-SNAPSHOT.jar into my install
> dir/share/hadoop/hdfs. Much faster than the full rebuild.
>
> -Todd
>
> On Wed, Feb 15, 2012 at 8:52 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>> Sriram,
>>
>> My package command didn't take as long with the same command -- where
>> do you see it getting stuck at most of the times? Which particular
>> target (right term?) seems to take the highest time to complete?
>>
>> On Wed, Feb 15, 2012 at 12:31 AM, Sriram Rao <[EMAIL PROTECTED]> wrote:
>>> Folks,
>>>
>>> The build takes forever:
>>>
>>>  mvn package -Pdist -DskipTests -Dtar
>>>
>>> ....
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD SUCCESS
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 6:25.399s
>>> [INFO] Finished at: Tue Feb 14 10:54:18 PST 2012
>>> [INFO] Final Memory: 56M/123M
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> This is on a mac with 8GB RAM.  (The above case involved 0-lines of code
>>> change; just making sure I had everything built!).
>>>
>>> Is there a faster way to get the thing to build?
>>>
>>> Sriram
>>
>>
>>
>> --
>> Harsh J
>> Customer Ops. Engineer
>> Cloudera | http://tiny.cloudera.com/about
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera

I've just done some tweaks in the POMs and I'm cutting the 8.30min full
dist build down to 3.45min. I'm removing all javadoc stuff from the dist
profile.

I think we should introduce a developer profile that does this.

Also, the 'mvn install -DskipTests', to get the JARs in your m2 cache is
taking 2mins. But we are building source JARs, the developer profile could
skip the generation of the source JARs, this would cut several seconds.

Finally, note that to just build all JARs 'mvn clean test -DskipTests' it
is taking 1.55mins.

If folk agree I'll open a JIRA to optmize the build based on these findings.

Thxs.

Alejandro

On Wed, Feb 15, 2012 at 1:02 PM, Daryn Sharp <[EMAIL PROTECTED]> wrote:

> Hi Sriram,
>
> You may want to try adding your source tree to Spotlight's privacy list,
> otherwise it's busy trying to index files as fast as the build is writing
> them.  If you are (un)fortunate enough to be running anti-virus sw, then it
> too is scurrying along behind your build checking every file.  That's
> enough to turn a normally silent MBP into a hair dryer.
>
> If you don't want to omit the entire project, because perhaps you want to
> use mdfind, then you can rename dirs to end with ".noindex".  The
> trick/hack is you can symlink the original dir name to the renamed
> dir.noindex.
>
> But yes, I think the build is way to slow to be productive when used as
> intended.  The fact that people have to use custom tricks to build & test
> in a "reasonable" amount of time is a symptom of a problem...
>
> Daryn
>
>
> On Feb 15, 2012, at 12:47 PM, Todd Lipcon wrote:
>
> > Hi Sriram,
> >
> > I also do -Dmaven.skip.javadoc=true usually and that cuts out a couple
> > minutes. Unless you need a tarball, you can also drop -Ptar.
> >
> > When I'm just making quick changes confined to the java code, I
> > usually just use "mvn -Dskiptests install" from within the hdfs
> > project, then manually cp the resulting
> > target/hadoop-hdfs-0.24.0-SNAPSHOT.jar into my install
> > dir/share/hadoop/hdfs. Much faster than the full rebuild.
> >
> > -Todd
> >
> > On Wed, Feb 15, 2012 at 8:52 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> >> Sriram,
> >>
> >> My package command didn't take as long with the same command -- where
> >> do you see it getting stuck at most of the times? Which particular
> >> target (right term?) seems to take the highest time to complete?
> >>
> >> On Wed, Feb 15, 2012 at 12:31 AM, Sriram Rao <[EMAIL PROTECTED]>
> wrote:
> >>> Folks,
> >>>
> >>> The build takes forever:
> >>>
> >>>  mvn package -Pdist -DskipTests -Dtar
> >>>
> >>> ....
> >>>
> >>> [INFO]
> >>>
> ------------------------------------------------------------------------
> >>> [INFO] BUILD SUCCESS
> >>> [INFO]
> >>>
> ------------------------------------------------------------------------
> >>> [INFO] Total time: 6:25.399s
> >>> [INFO] Finished at: Tue Feb 14 10:54:18 PST 2012
> >>> [INFO] Final Memory: 56M/123M
> >>> [INFO]
> >>>
> ------------------------------------------------------------------------
> >>>
> >>> This is on a mac with 8GB RAM.  (The above case involved 0-lines of
> code
> >>> change; just making sure I had everything built!).
> >>>
> >>> Is there a faster way to get the thing to build?
> >>>
> >>> Sriram
> >>
> >>
> >>
> >> --
> >> Harsh J
> >> Customer Ops. Engineer
> >> Cloudera | http://tiny.cloudera.com/about
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>
>

Wow, nice!  How long is a rebuild with no changes?

I'd be delighted if there was a dev profile that just did the bare minimum to produce a working env.

Daryn
On Feb 16, 2012, at 3:25 PM, Alejandro Abdelnur wrote:

> I've just done some tweaks in the POMs and I'm cutting the 8.30min full
> dist build down to 3.45min. I'm removing all javadoc stuff from the dist
> profile.
>
> I think we should introduce a developer profile that does this.
>
> Also, the 'mvn install -DskipTests', to get the JARs in your m2 cache is
> taking 2mins. But we are building source JARs, the developer profile could
> skip the generation of the source JARs, this would cut several seconds.
>
> Finally, note that to just build all JARs 'mvn clean test -DskipTests' it
> is taking 1.55mins.
>
> If folk agree I'll open a JIRA to optmize the build based on these findings.
>
> Thxs.
>
> Alejandro
>
> On Wed, Feb 15, 2012 at 1:02 PM, Daryn Sharp <[EMAIL PROTECTED]> wrote:
>
>> Hi Sriram,
>>
>> You may want to try adding your source tree to Spotlight's privacy list,
>> otherwise it's busy trying to index files as fast as the build is writing
>> them.  If you are (un)fortunate enough to be running anti-virus sw, then it
>> too is scurrying along behind your build checking every file.  That's
>> enough to turn a normally silent MBP into a hair dryer.
>>
>> If you don't want to omit the entire project, because perhaps you want to
>> use mdfind, then you can rename dirs to end with ".noindex".  The
>> trick/hack is you can symlink the original dir name to the renamed
>> dir.noindex.
>>
>> But yes, I think the build is way to slow to be productive when used as
>> intended.  The fact that people have to use custom tricks to build & test
>> in a "reasonable" amount of time is a symptom of a problem...
>>
>> Daryn
>>
>>
>> On Feb 15, 2012, at 12:47 PM, Todd Lipcon wrote:
>>
>>> Hi Sriram,
>>>
>>> I also do -Dmaven.skip.javadoc=true usually and that cuts out a couple
>>> minutes. Unless you need a tarball, you can also drop -Ptar.
>>>
>>> When I'm just making quick changes confined to the java code, I
>>> usually just use "mvn -Dskiptests install" from within the hdfs
>>> project, then manually cp the resulting
>>> target/hadoop-hdfs-0.24.0-SNAPSHOT.jar into my install
>>> dir/share/hadoop/hdfs. Much faster than the full rebuild.
>>>
>>> -Todd
>>>
>>> On Wed, Feb 15, 2012 at 8:52 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>>> Sriram,
>>>>
>>>> My package command didn't take as long with the same command -- where
>>>> do you see it getting stuck at most of the times? Which particular
>>>> target (right term?) seems to take the highest time to complete?
>>>>
>>>> On Wed, Feb 15, 2012 at 12:31 AM, Sriram Rao <[EMAIL PROTECTED]>
>> wrote:
>>>>> Folks,
>>>>>
>>>>> The build takes forever:
>>>>>
>>>>> mvn package -Pdist -DskipTests -Dtar
>>>>>
>>>>> ....
>>>>>
>>>>> [INFO]
>>>>>
>> ------------------------------------------------------------------------
>>>>> [INFO] BUILD SUCCESS
>>>>> [INFO]
>>>>>
>> ------------------------------------------------------------------------
>>>>> [INFO] Total time: 6:25.399s
>>>>> [INFO] Finished at: Tue Feb 14 10:54:18 PST 2012
>>>>> [INFO] Final Memory: 56M/123M
>>>>> [INFO]
>>>>>
>> ------------------------------------------------------------------------
>>>>>
>>>>> This is on a mac with 8GB RAM.  (The above case involved 0-lines of
>> code
>>>>> change; just making sure I had everything built!).
>>>>>
>>>>> Is there a faster way to get the thing to build?
>>>>>
>>>>> Sriram
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>> Customer Ops. Engineer
>>>> Cloudera | http://tiny.cloudera.com/about
>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>
>>

A 'mvn package -DskipTests -Pdist'  (developer) with no changes takes
3.35mins

The biggest offenders are:

[INFO] Apache Hadoop Common .............................. SUCCESS [43.299s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [54.776s]
[INFO] hadoop-yarn-api ................................... SUCCESS [15.466s]
[INFO] hadoop-mapreduce .................................. SUCCESS [16.154s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [9.657s]

Everybody else is below 5secs

A 'mvn test -DskipTests' with no changes 1.05mins (this should go down
making the some codegen to be smart instead just compiling again)

The biggest offenders are:

[INFO] Apache Hadoop Common .............................. SUCCESS [9.121s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [20.887s]
[INFO] hadoop-yarn-api ................................... SUCCESS [6.664s]

A few 2sec modules and the rest below 1sec

Thanks.

Alejandro
On Thu, Feb 16, 2012 at 2:13 PM, Daryn Sharp <[EMAIL PROTECTED]> wrote:

> Wow, nice!  How long is a rebuild with no changes?
>
> I'd be delighted if there was a dev profile that just did the bare minimum
> to produce a working env.
>
> Daryn
>
>
> On Feb 16, 2012, at 3:25 PM, Alejandro Abdelnur wrote:
>
> > I've just done some tweaks in the POMs and I'm cutting the 8.30min full
> > dist build down to 3.45min. I'm removing all javadoc stuff from the dist
> > profile.
> >
> > I think we should introduce a developer profile that does this.
> >
> > Also, the 'mvn install -DskipTests', to get the JARs in your m2 cache is
> > taking 2mins. But we are building source JARs, the developer profile
> could
> > skip the generation of the source JARs, this would cut several seconds.
> >
> > Finally, note that to just build all JARs 'mvn clean test -DskipTests' it
> > is taking 1.55mins.
> >
> > If folk agree I'll open a JIRA to optmize the build based on these
> findings.
> >
> > Thxs.
> >
> > Alejandro
> >
> > On Wed, Feb 15, 2012 at 1:02 PM, Daryn Sharp <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hi Sriram,
> >>
> >> You may want to try adding your source tree to Spotlight's privacy list,
> >> otherwise it's busy trying to index files as fast as the build is
> writing
> >> them.  If you are (un)fortunate enough to be running anti-virus sw,
> then it
> >> too is scurrying along behind your build checking every file.  That's
> >> enough to turn a normally silent MBP into a hair dryer.
> >>
> >> If you don't want to omit the entire project, because perhaps you want
> to
> >> use mdfind, then you can rename dirs to end with ".noindex".  The
> >> trick/hack is you can symlink the original dir name to the renamed
> >> dir.noindex.
> >>
> >> But yes, I think the build is way to slow to be productive when used as
> >> intended.  The fact that people have to use custom tricks to build &
> test
> >> in a "reasonable" amount of time is a symptom of a problem...
> >>
> >> Daryn
> >>
> >>
> >> On Feb 15, 2012, at 12:47 PM, Todd Lipcon wrote:
> >>
> >>> Hi Sriram,
> >>>
> >>> I also do -Dmaven.skip.javadoc=true usually and that cuts out a couple
> >>> minutes. Unless you need a tarball, you can also drop -Ptar.
> >>>
> >>> When I'm just making quick changes confined to the java code, I
> >>> usually just use "mvn -Dskiptests install" from within the hdfs
> >>> project, then manually cp the resulting
> >>> target/hadoop-hdfs-0.24.0-SNAPSHOT.jar into my install
> >>> dir/share/hadoop/hdfs. Much faster than the full rebuild.
> >>>
> >>> -Todd
> >>>
> >>> On Wed, Feb 15, 2012 at 8:52 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> >>>> Sriram,
> >>>>
> >>>> My package command didn't take as long with the same command -- where
> >>>> do you see it getting stuck at most of the times? Which particular
> >>>> target (right term?) seems to take the highest time to complete?
> >>>>
> >>>> On Wed, Feb 15, 2012 at 12:31 AM, Sriram Rao <[EMAIL PROTECTED]>
> >> wrote:
> >>>>> Folks,
> >>>>>
> >>>>> The build takes forever:

Very nice improvement!  How long is "mvn install -DskipTests -Dmaven.javadoc.skip"?  Assuming that's fast, is there possibly a way to allow hadoop to run out of the checkedout tree?  Maybe through the CLASSPATH, or by subprojects copying the built jars into the dist area?  The ideal would be to run a full maven build, followed by just a mvn build within a specific subproject.

Daryn
On Feb 16, 2012, at 6:15 PM, Alejandro Abdelnur wrote:

> A 'mvn package -DskipTests -Pdist'  (developer) with no changes takes
> 3.35mins
>
> The biggest offenders are:
>
> [INFO] Apache Hadoop Common .............................. SUCCESS [43.299s]
> [INFO] Apache Hadoop HDFS ................................ SUCCESS [54.776s]
> [INFO] hadoop-yarn-api ................................... SUCCESS [15.466s]
> [INFO] hadoop-mapreduce .................................. SUCCESS [16.154s]
> [INFO] Apache Hadoop Distributed Copy .................... SUCCESS [9.657s]
>
> Everybody else is below 5secs
>
> A 'mvn test -DskipTests' with no changes 1.05mins (this should go down
> making the some codegen to be smart instead just compiling again)
>
> The biggest offenders are:
>
> [INFO] Apache Hadoop Common .............................. SUCCESS [9.121s]
> [INFO] Apache Hadoop HDFS ................................ SUCCESS [20.887s]
> [INFO] hadoop-yarn-api ................................... SUCCESS [6.664s]
>
> A few 2sec modules and the rest below 1sec
>
> Thanks.
>
> Alejandro
>
>
> On Thu, Feb 16, 2012 at 2:13 PM, Daryn Sharp <[EMAIL PROTECTED]> wrote:
>
>> Wow, nice!  How long is a rebuild with no changes?
>>
>> I'd be delighted if there was a dev profile that just did the bare minimum
>> to produce a working env.
>>
>> Daryn
>>
>>
>> On Feb 16, 2012, at 3:25 PM, Alejandro Abdelnur wrote:
>>
>>> I've just done some tweaks in the POMs and I'm cutting the 8.30min full
>>> dist build down to 3.45min. I'm removing all javadoc stuff from the dist
>>> profile.
>>>
>>> I think we should introduce a developer profile that does this.
>>>
>>> Also, the 'mvn install -DskipTests', to get the JARs in your m2 cache is
>>> taking 2mins. But we are building source JARs, the developer profile
>> could
>>> skip the generation of the source JARs, this would cut several seconds.
>>>
>>> Finally, note that to just build all JARs 'mvn clean test -DskipTests' it
>>> is taking 1.55mins.
>>>
>>> If folk agree I'll open a JIRA to optmize the build based on these
>> findings.
>>>
>>> Thxs.
>>>
>>> Alejandro
>>>
>>> On Wed, Feb 15, 2012 at 1:02 PM, Daryn Sharp <[EMAIL PROTECTED]>
>> wrote:
>>>
>>>> Hi Sriram,
>>>>
>>>> You may want to try adding your source tree to Spotlight's privacy list,
>>>> otherwise it's busy trying to index files as fast as the build is
>> writing
>>>> them.  If you are (un)fortunate enough to be running anti-virus sw,
>> then it
>>>> too is scurrying along behind your build checking every file.  That's
>>>> enough to turn a normally silent MBP into a hair dryer.
>>>>
>>>> If you don't want to omit the entire project, because perhaps you want
>> to
>>>> use mdfind, then you can rename dirs to end with ".noindex".  The
>>>> trick/hack is you can symlink the original dir name to the renamed
>>>> dir.noindex.
>>>>
>>>> But yes, I think the build is way to slow to be productive when used as
>>>> intended.  The fact that people have to use custom tricks to build &
>> test
>>>> in a "reasonable" amount of time is a symptom of a problem...
>>>>
>>>> Daryn
>>>>
>>>>
>>>> On Feb 15, 2012, at 12:47 PM, Todd Lipcon wrote:
>>>>
>>>>> Hi Sriram,
>>>>>
>>>>> I also do -Dmaven.skip.javadoc=true usually and that cuts out a couple
>>>>> minutes. Unless you need a tarball, you can also drop -Ptar.
>>>>>
>>>>> When I'm just making quick changes confined to the java code, I
>>>>> usually just use "mvn -Dskiptests install" from within the hdfs
>>>>> project, then manually cp the resulting
>>>>> target/hadoop-hdfs-0.24.0-SNAPSHOT.jar into my install