Andy:
I agree that pre-commit build environment left something to be desired.
It seems not all machines are equally bad.
Hopefully Giri or someone else can take a look at the build machines.

On Dec 3, 2011, at 9:26 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:

> First with the file handles, now with nproc ulimits:
>
>> java.lang.OutOfMemoryError: unable to create new native threadWho configures the test systems? "HadoopQA" output is not useful. It's just polluting JIRAs and should be shut off.
>
>    - Andy
>
>
>
>
> ----- Original Message -----
>> From: Hadoop QA (Commented) (JIRA) <[EMAIL PROTECTED]>
>> To: [EMAIL PROTECTED]
>> Cc:
>> Sent: Sunday, December 4, 2011 1:22 PM
>> Subject: [jira] [Commented] (HBASE-4944) Optionally verify bulk loaded HFiles
>>
>>
>>     [
>> https://issues.apache.org/jira/browse/HBASE-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162291#comment-13162291
>> ]
>>
>> Hadoop QA commented on HBASE-4944:
>> ----------------------------------
>>
>> -1 overall.  Here are the results of testing the latest attachment
>>   http://issues.apache.org/jira/secure/attachment/12506018/HBASE-4944-v3.patch
>>   against trunk revision .
>>
>>     +1 @author.  The patch does not contain any @author tags.
>>
>>     -1 tests included.  The patch doesn't appear to include any new or
>> modified tests.
>>                         Please justify why no new tests are needed for this
>> patch.
>>                         Also please list what manual steps were performed to
>> verify this patch.
>>
>>     -1 javadoc.  The javadoc tool appears to have generated -160 warning
>> messages.
>>
>>     +1 javac.  The applied patch does not increase the total number of javac
>> compiler warnings.
>>
>>     -1 findbugs.  The patch appears to introduce 71 new Findbugs (version 1.3.9)
>> warnings.
>>
>>     +1 release audit.  The applied patch does not increase the total number of
>> release audit warnings.
>>
>>      -1 core tests.  The patch failed these unit tests:
>>                        org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
>>                  
>> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery
>>                   org.apache.hadoop.hbase.TestDrainingServer
>>                   org.apache.hadoop.hbase.TestFullLogReconstruction
>>                   org.apache.hadoop.hbase.mapreduce.TestImportTsv
>>                   org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
>>
>> Test results:
>> https://builds.apache.org/job/PreCommit-HBASE-Build/437//testReport/
>> Findbugs warnings:
>> https://builds.apache.org/job/PreCommit-HBASE-Build/437//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
>> Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/437//console
>>
>> This message is automatically generated.
>>                
>>> Optionally verify bulk loaded HFiles
>>> ------------------------------------
>>>
>>>                  Key: HBASE-4944
>>>                  URL: https://issues.apache.org/jira/browse/HBASE-4944
>>>              Project: HBase
>>>           Issue Type: Improvement
>>>           Components: regionserver
>>>     Affects Versions: 0.92.0, 0.94.0, 0.90.5
>>>             Reporter: Andrew Purtell
>>>             Priority: Minor
>>>          Attachments: HBASE-4944-v2.patch, HBASE-4944-v3.patch
>>>
>>>
>>> We rely on users to produce properly formatted HFiles for bulk import.
>> Attached patch adds an optional code path, toggled by a configuration property,
>> that verifies the HFile under consideration for import is properly sorted. The
>> default maintains the current behavior, which does not scan the file for
>> correctness.
>>> Patch is against trunk but can apply against all active branches.
>>
>> --
>> This message is automatically generated by JIRA.
>> If you think it was sent incorrectly, please contact your JIRA administrators:
>> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa

I discovered recently that hat some tests leak threads, and I got exactly
this message. For example, they close the region but not the hlog linked to
the region. It does not explain why it worked on most env but not on
hadoop-qa, it would be interesting to know the exact limits on hadoop-qa.

On Sun, Dec 4, 2011 at 5:44 PM, <[EMAIL PROTECTED]> wrote:

> Andy:
> I agree that pre-commit build environment left something to be desired.
> It seems not all machines are equally bad.
> Hopefully Giri or someone else can take a look at the build machines.
>
>
>
> On Dec 3, 2011, at 9:26 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
>
> > First with the file handles, now with nproc ulimits:
> >
> >> java.lang.OutOfMemoryError: unable to create new native threadWho
> configures the test systems? "HadoopQA" output is not useful. It's just
> polluting JIRAs and should be shut off.
> >
> >    - Andy
> >
> >
> >
> >
> > ----- Original Message -----
> >> From: Hadoop QA (Commented) (JIRA) <[EMAIL PROTECTED]>
> >> To: [EMAIL PROTECTED]
> >> Cc:
> >> Sent: Sunday, December 4, 2011 1:22 PM
> >> Subject: [jira] [Commented] (HBASE-4944) Optionally verify bulk loaded
> HFiles
> >>
> >>
> >>     [
> >>
> https://issues.apache.org/jira/browse/HBASE-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162291#comment-13162291
> >> ]
> >>
> >> Hadoop QA commented on HBASE-4944:
> >> ----------------------------------
> >>
> >> -1 overall.  Here are the results of testing the latest attachment
> >>
> http://issues.apache.org/jira/secure/attachment/12506018/HBASE-4944-v3.patch
> >>   against trunk revision .
> >>
> >>     +1 @author.  The patch does not contain any @author tags.
> >>
> >>     -1 tests included.  The patch doesn't appear to include any new or
> >> modified tests.
> >>                         Please justify why no new tests are needed for
> this
> >> patch.
> >>                         Also please list what manual steps were
> performed to
> >> verify this patch.
> >>
> >>     -1 javadoc.  The javadoc tool appears to have generated -160 warning
> >> messages.
> >>
> >>     +1 javac.  The applied patch does not increase the total number of
> javac
> >> compiler warnings.
> >>
> >>     -1 findbugs.  The patch appears to introduce 71 new Findbugs
> (version 1.3.9)
> >> warnings.
> >>
> >>     +1 release audit.  The applied patch does not increase the total
> number of
> >> release audit warnings.
> >>
> >>      -1 core tests.  The patch failed these unit tests:
> >>
>  org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
> >>
> >> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery
> >>                   org.apache.hadoop.hbase.TestDrainingServer
> >>                   org.apache.hadoop.hbase.TestFullLogReconstruction
> >>                   org.apache.hadoop.hbase.mapreduce.TestImportTsv
> >>                   org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
> >>
> >> Test results:
> >> https://builds.apache.org/job/PreCommit-HBASE-Build/437//testReport/
> >> Findbugs warnings:
> >>
> https://builds.apache.org/job/PreCommit-HBASE-Build/437//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
> >> Console output:
> https://builds.apache.org/job/PreCommit-HBASE-Build/437//console
> >>
> >> This message is automatically generated.
> >>
> >>> Optionally verify bulk loaded HFiles
> >>> ------------------------------------
> >>>
> >>>                  Key: HBASE-4944
> >>>                  URL: https://issues.apache.org/jira/browse/HBASE-4944
> >>>              Project: HBase
> >>>           Issue Type: Improvement
> >>>           Components: regionserver
> >>>     Affects Versions: 0.92.0, 0.94.0, 0.90.5
> >>>             Reporter: Andrew Purtell
> >>>             Priority: Minor
> >>>          Attachments: HBASE-4944-v2.patch, HBASE-4944-v3.patch
> >>>
> >>>
> >>> We rely on users to produce properly formatted HFiles for bulk import.
> >> Attached patch adds an optional code path, toggled by a configuration

I think for non-trivial fix / patch, contributors should provide test suite result.
This would reduce the noise mentioned by Andy and give us higher confidence before committing.

Cheers

On Dec 4, 2011, at 9:57 AM, N Keywal <[EMAIL PROTECTED]> wrote:

> I discovered recently that hat some tests leak threads, and I got exactly
> this message. For example, they close the region but not the hlog linked to
> the region. It does not explain why it worked on most env but not on
> hadoop-qa, it would be interesting to know the exact limits on hadoop-qa.
>
> On Sun, Dec 4, 2011 at 5:44 PM, <[EMAIL PROTECTED]> wrote:
>
>> Andy:
>> I agree that pre-commit build environment left something to be desired.
>> It seems not all machines are equally bad.
>> Hopefully Giri or someone else can take a look at the build machines.
>>
>>
>>
>> On Dec 3, 2011, at 9:26 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
>>
>>> First with the file handles, now with nproc ulimits:
>>>
>>>> java.lang.OutOfMemoryError: unable to create new native threadWho
>> configures the test systems? "HadoopQA" output is not useful. It's just
>> polluting JIRAs and should be shut off.
>>>
>>>   - Andy
>>>
>>>
>>>
>>>
>>> ----- Original Message -----
>>>> From: Hadoop QA (Commented) (JIRA) <[EMAIL PROTECTED]>
>>>> To: [EMAIL PROTECTED]
>>>> Cc:
>>>> Sent: Sunday, December 4, 2011 1:22 PM
>>>> Subject: [jira] [Commented] (HBASE-4944) Optionally verify bulk loaded
>> HFiles
>>>>
>>>>
>>>>    [
>>>>
>> https://issues.apache.org/jira/browse/HBASE-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162291#comment-13162291
>>>> ]
>>>>
>>>> Hadoop QA commented on HBASE-4944:
>>>> ----------------------------------
>>>>
>>>> -1 overall.  Here are the results of testing the latest attachment
>>>>
>> http://issues.apache.org/jira/secure/attachment/12506018/HBASE-4944-v3.patch
>>>>  against trunk revision .
>>>>
>>>>    +1 @author.  The patch does not contain any @author tags.
>>>>
>>>>    -1 tests included.  The patch doesn't appear to include any new or
>>>> modified tests.
>>>>                        Please justify why no new tests are needed for
>> this
>>>> patch.
>>>>                        Also please list what manual steps were
>> performed to
>>>> verify this patch.
>>>>
>>>>    -1 javadoc.  The javadoc tool appears to have generated -160 warning
>>>> messages.
>>>>
>>>>    +1 javac.  The applied patch does not increase the total number of
>> javac
>>>> compiler warnings.
>>>>
>>>>    -1 findbugs.  The patch appears to introduce 71 new Findbugs
>> (version 1.3.9)
>>>> warnings.
>>>>
>>>>    +1 release audit.  The applied patch does not increase the total
>> number of
>>>> release audit warnings.
>>>>
>>>>     -1 core tests.  The patch failed these unit tests:
>>>>
>> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
>>>>
>>>> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery
>>>>                  org.apache.hadoop.hbase.TestDrainingServer
>>>>                  org.apache.hadoop.hbase.TestFullLogReconstruction
>>>>                  org.apache.hadoop.hbase.mapreduce.TestImportTsv
>>>>                  org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
>>>>
>>>> Test results:
>>>> https://builds.apache.org/job/PreCommit-HBASE-Build/437//testReport/
>>>> Findbugs warnings:
>>>>
>> https://builds.apache.org/job/PreCommit-HBASE-Build/437//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
>>>> Console output:
>> https://builds.apache.org/job/PreCommit-HBASE-Build/437//console
>>>>
>>>> This message is automatically generated.
>>>>
>>>>> Optionally verify bulk loaded HFiles
>>>>> ------------------------------------
>>>>>
>>>>>                 Key: HBASE-4944
>>>>>                 URL: https://issues.apache.org/jira/browse/HBASE-4944
>>>>>             Project: HBase
>>>>>          Issue Type: Improvement
>>>>>          Components: regionserver
>>>>>    Affects Versions: 0.92.0, 0.94.0, 0.90.5