Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # dev - .23 compile times


+
Sriram Rao 2012-02-14, 19:01
+
Harsh J 2012-02-15, 16:52
+
Todd Lipcon 2012-02-15, 18:47
+
Daryn Sharp 2012-02-15, 21:02
+
Alejandro Abdelnur 2012-02-16, 21:25
Copy link to this message
-
Re: .23 compile times
Daryn Sharp 2012-02-16, 22:13
Wow, nice!  How long is a rebuild with no changes?

I'd be delighted if there was a dev profile that just did the bare minimum to produce a working env.

Daryn
On Feb 16, 2012, at 3:25 PM, Alejandro Abdelnur wrote:

> I've just done some tweaks in the POMs and I'm cutting the 8.30min full
> dist build down to 3.45min. I'm removing all javadoc stuff from the dist
> profile.
>
> I think we should introduce a developer profile that does this.
>
> Also, the 'mvn install -DskipTests', to get the JARs in your m2 cache is
> taking 2mins. But we are building source JARs, the developer profile could
> skip the generation of the source JARs, this would cut several seconds.
>
> Finally, note that to just build all JARs 'mvn clean test -DskipTests' it
> is taking 1.55mins.
>
> If folk agree I'll open a JIRA to optmize the build based on these findings.
>
> Thxs.
>
> Alejandro
>
> On Wed, Feb 15, 2012 at 1:02 PM, Daryn Sharp <[EMAIL PROTECTED]> wrote:
>
>> Hi Sriram,
>>
>> You may want to try adding your source tree to Spotlight's privacy list,
>> otherwise it's busy trying to index files as fast as the build is writing
>> them.  If you are (un)fortunate enough to be running anti-virus sw, then it
>> too is scurrying along behind your build checking every file.  That's
>> enough to turn a normally silent MBP into a hair dryer.
>>
>> If you don't want to omit the entire project, because perhaps you want to
>> use mdfind, then you can rename dirs to end with ".noindex".  The
>> trick/hack is you can symlink the original dir name to the renamed
>> dir.noindex.
>>
>> But yes, I think the build is way to slow to be productive when used as
>> intended.  The fact that people have to use custom tricks to build & test
>> in a "reasonable" amount of time is a symptom of a problem...
>>
>> Daryn
>>
>>
>> On Feb 15, 2012, at 12:47 PM, Todd Lipcon wrote:
>>
>>> Hi Sriram,
>>>
>>> I also do -Dmaven.skip.javadoc=true usually and that cuts out a couple
>>> minutes. Unless you need a tarball, you can also drop -Ptar.
>>>
>>> When I'm just making quick changes confined to the java code, I
>>> usually just use "mvn -Dskiptests install" from within the hdfs
>>> project, then manually cp the resulting
>>> target/hadoop-hdfs-0.24.0-SNAPSHOT.jar into my install
>>> dir/share/hadoop/hdfs. Much faster than the full rebuild.
>>>
>>> -Todd
>>>
>>> On Wed, Feb 15, 2012 at 8:52 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>>> Sriram,
>>>>
>>>> My package command didn't take as long with the same command -- where
>>>> do you see it getting stuck at most of the times? Which particular
>>>> target (right term?) seems to take the highest time to complete?
>>>>
>>>> On Wed, Feb 15, 2012 at 12:31 AM, Sriram Rao <[EMAIL PROTECTED]>
>> wrote:
>>>>> Folks,
>>>>>
>>>>> The build takes forever:
>>>>>
>>>>> mvn package -Pdist -DskipTests -Dtar
>>>>>
>>>>> ....
>>>>>
>>>>> [INFO]
>>>>>
>> ------------------------------------------------------------------------
>>>>> [INFO] BUILD SUCCESS
>>>>> [INFO]
>>>>>
>> ------------------------------------------------------------------------
>>>>> [INFO] Total time: 6:25.399s
>>>>> [INFO] Finished at: Tue Feb 14 10:54:18 PST 2012
>>>>> [INFO] Final Memory: 56M/123M
>>>>> [INFO]
>>>>>
>> ------------------------------------------------------------------------
>>>>>
>>>>> This is on a mac with 8GB RAM.  (The above case involved 0-lines of
>> code
>>>>> change; just making sure I had everything built!).
>>>>>
>>>>> Is there a faster way to get the thing to build?
>>>>>
>>>>> Sriram
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>> Customer Ops. Engineer
>>>> Cloudera | http://tiny.cloudera.com/about
>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>
>>
+
Alejandro Abdelnur 2012-02-17, 00:15
+
Daryn Sharp 2012-02-17, 15:03