Hi guys,

Every build in recent history run by the PreCommit job [0] has failed. For
some of them, this is the usual suspect of a patch not applying cleanly.
But for many, the issue looks like a simple configuration issue. Case in
point, this one [1] looks like Jenkins simply failed to read a value from
the JIRA. Scroll to the very bottom, you'll see what I mean. Is there a way
to set a default value for this field on the Jenkins side (something like
empty string, perhaps)?

A CI system is only as good as it is reliable...

Thanks,
Nick

[0]: https://builds.apache.org/job/PreCommit-HBASE-Build/
[1]: https://builds.apache.org/job/PreCommit-HBASE-Build/3526/consoleText

On Thu, Dec 13, 2012 at 10:24 AM, Nick Dimiduk <[EMAIL PROTECTED]> wrote:

> Hi guys,
>
> Every build in recent history run by the PreCommit job [0] has failed. For
> some of them, this is the usual suspect of a patch not applying cleanly.
> But for many, the issue looks like a simple configuration issue. Case in
> point, this one [1] looks like Jenkins simply failed to read a value from
> the JIRA. Scroll to the very bottom, you'll see what I mean. Is there a way
> to set a default value for this field on the Jenkins side (something like
> empty string, perhaps)?
>

Thanks for trying to take a look Nick.  If you need admin privileges to
poke around, I can grant them to you....
St.Ack

For #3526, I think it's marked wrong because one test timeouted. This is
not really well managed in surefire today.
I actually hope that the next version (2.13) will improve this.

I think that we also have builds marked as bad with this:

    {color:red}-1 javadoc{color}.  The javadoc tool appears to have
generated 104 warning messages.
    {color:red}-1 findbugs{color}.  The patch appears to introduce 23
new Findbugs (version 1.3.9) warnings.

And IIRC, it's actually bugs in the build scripts.

For sure, any improvement here would be great.

On Thu, Dec 13, 2012 at 7:24 PM, Nick Dimiduk <[EMAIL PROTECTED]> wrote:

> Hi guys,
>
> Every build in recent history run by the PreCommit job [0] has failed. For
> some of them, this is the usual suspect of a patch not applying cleanly.
> But for many, the issue looks like a simple configuration issue. Case in
> point, this one [1] looks like Jenkins simply failed to read a value from
> the JIRA. Scroll to the very bottom, you'll see what I mean. Is there a way
> to set a default value for this field on the Jenkins side (something like
> empty string, perhaps)?
>
> A CI system is only as good as it is reliable...
>
> Thanks,
> Nick
>
> [0]: https://builds.apache.org/job/PreCommit-HBASE-Build/
> [1]: https://builds.apache.org/job/PreCommit-HBASE-Build/3526/consoleText
>

@Stack: Hey now, let's not get to hasty!

@Nicolas,

Looking at the output of #3526 [0], where does this bash
script /tmp/hudson4033738860406301156.sh come from?

What does the -DHBasePatchProcess flag do? A `git grep` yields no results.

The step "Determining number of patched javadoc warnings." would be more
useful if it ran javadoc on unpatched trunk and compared results to produce
the number of new warnings produced. It claims to do something similar for
javac warnings (though I don't believe it).

Tests for hbase-server failed with no clear reason produced. Perhaps
running secondPartTestsExecution in serial will reveal the failure/timeout?

Whatever script is handling "Recording test results" could use the default
value i mentioned in my initial comment.

[0]: https://builds.apache.org/job/PreCommit-HBASE-Build/3526/consoleText

On Thu, Dec 13, 2012 at 10:44 AM, Nicolas Liochon <[EMAIL PROTECTED]> wrote:

> For #3526, I think it's marked wrong because one test timeouted. This is
> not really well managed in surefire today.
> I actually hope that the next version (2.13) will improve this.
>
> I think that we also have builds marked as bad with this:
>
>     {color:red}-1 javadoc{color}.  The javadoc tool appears to have
> generated 104 warning messages.
>     {color:red}-1 findbugs{color}.  The patch appears to introduce 23
> new Findbugs (version 1.3.9) warnings.
>
> And IIRC, it's actually bugs in the build scripts.
>
> For sure, any improvement here would be great.
>
>
>
> On Thu, Dec 13, 2012 at 7:24 PM, Nick Dimiduk <[EMAIL PROTECTED]> wrote:
>
> > Hi guys,
> >
> > Every build in recent history run by the PreCommit job [0] has failed.
> For
> > some of them, this is the usual suspect of a patch not applying cleanly.
> > But for many, the issue looks like a simple configuration issue. Case in
> > point, this one [1] looks like Jenkins simply failed to read a value from
> > the JIRA. Scroll to the very bottom, you'll see what I mean. Is there a
> way
> > to set a default value for this field on the Jenkins side (something like
> > empty string, perhaps)?
> >
> > A CI system is only as good as it is reliable...
> >
> > Thanks,
> > Nick
> >
> > [0]: https://builds.apache.org/job/PreCommit-HBASE-Build/
> > [1]:
> https://builds.apache.org/job/PreCommit-HBASE-Build/3526/consoleText
> >
>

On Thu, Dec 13, 2012 at 11:56 AM, Nick Dimiduk <[EMAIL PROTECTED]> wrote:

> @Stack: Hey now, let's not get to hasty!
>
>

Just sent you all my life savings, $50.00, and signed over to you my first
and second born.

> @Nicolas,
>
> Looking at the output of #3526 [0], where does this bash
> script /tmp/hudson4033738860406301156.sh come from?
>
>
It looks like this is what it is running:
#!/bin/bash
hostname
uname -a
ulimit -a
set +x
source ${WORKSPACE}/nightly/hudsonEnv.sh
export ANT_OPTS=-Xmx2048m
mkdir -p ${WORKSPACE}/patchprocess
cd ${TRUNK}

/bin/bash ${TRUNK}/dev-support/test-patch.sh \
        --jenkins \
        --patch-dir=${WORKSPACE}/patchprocess \
        --basedir=${TRUNK} \
        --support-dir=/home/jenkins/buildSupport \
        --ps-cmd=/bin/ps \
        --wget-cmd=/usr/bin/wget \
        --svn-cmd=/usr/bin/svn \
        --grep-cmd=/bin/grep \
        --patch-cmd=/usr/bin/patch \
        --jira-cmd=/home/jenkins/tools/jiracli/latest/jira \
        --jira-password=4hadoopqa \
        --findbugs-home=/home/jenkins/tools/findbugs/latest \
        --forrest-home=/home/jenkins/tools/forrest/latest \
        --eclipse--home=/home/jenkins/tools/eclipse/latest \
        HBASE-${ISSUE_NUM}

This is the content of a textedit box that you can fill in configuring the
build in jenkins.  Here is the help for the textbox:

"Runs a shell script (defaults to sh, but this is configurable) for
building the project. The script will be run with the workspace as the
current directory. Type in the contents of your shell script. If your shell
script has no header line like #!/bin/sh —, then the shell configured
system-wide will be used, but you can also use the header line to write
script in another language (like #!/bin/perl) or control the options that
shell uses.

By default, the shell will be invoked with the "-ex" option. So all of the
commands are printed before being executed, and the build is considered a
failure if any of the commands exits with a non-zero exit code. Again, add
the #!/bin/... line to change this behavior.

As a best practice, try not to put a long shell script in here. Instead,
consider adding the shell script in SCM and simply call that shell script
from Jenkins (via bash -ex myscript.sh or something like that), so that you
can track changes in your shell script."

> What does the -DHBasePatchProcess flag do? A `git grep` yields no results.
>
>

I don't know.

I see it made up in ./dev-support/test-patch.sh

 3   $MVN clean test -DskipTests -D${PROJECT_NAME}PatchProcess >
$PATCH_DIR/trunkJavacWarnings.txt 2>&1

> The step "Determining number of patched javadoc warnings." would be more
> useful if it ran javadoc on unpatched trunk and compared results to produce
> the number of new warnings produced. It claims to do something similar for
> javac warnings (though I don't believe it).
>
>
Yes.

> Tests for hbase-server failed with no clear reason produced. Perhaps
> running secondPartTestsExecution in serial will reveal the failure/timeout?
>
>

That's be sweet.

> Whatever script is handling "Recording test results" could use the default
> value i mentioned in my initial comment.
>
>

Is it the return out of the function inside ./dev-support/test-patch.sh
called submitJiraComment?

St.Ack
> [0]: https://builds.apache.org/job/PreCommit-HBASE-Build/3526/consoleText
>
> On Thu, Dec 13, 2012 at 10:44 AM, Nicolas Liochon <[EMAIL PROTECTED]>
> wrote:
>
> > For #3526, I think it's marked wrong because one test timeouted. This is
> > not really well managed in surefire today.
> > I actually hope that the next version (2.13) will improve this.
> >
> > I think that we also have builds marked as bad with this:
> >
> >     {color:red}-1 javadoc{color}.  The javadoc tool appears to have
> > generated 104 warning messages.
> >     {color:red}-1 findbugs{color}.  The patch appears to introduce 23
> > new Findbugs (version 1.3.9) warnings.
> >
> > And IIRC, it's actually bugs in the build scripts.
> >

On Thu, Dec 13, 2012 at 1:19 PM, Stack <[EMAIL PROTECTED]> wrote:

> > What does the -DHBasePatchProcess flag do? A `git grep` yields no
> results.
>
> I don't know.
>
> I see it made up in ./dev-support/test-patch.sh
>
>  3   $MVN clean test -DskipTests -D${PROJECT_NAME}PatchProcess >
> $PATCH_DIR/trunkJavacWarnings.txt 2>&1
It's used on line 614 to find and kill any "rogue build processes from the
last attempt." Doesn't that mean any other builds running concurrently on
this host will be assassinated by the new build? Does Jenkins ensure only
one HBasePatchProcess runs on a given host at a time?

> The step "Determining number of patched javadoc warnings." would be more
> > useful if it ran javadoc on unpatched trunk and compared results to
> produce
> > the number of new warnings produced. It claims to do something similar
> for
> > javac warnings (though I don't believe it).
>
> Yes.
>

I've opened a ticket to add this feature: HBASE-7349. I believe it can also
deprecate the variable OK_JAVADOC_WARNINGS.

> Tests for hbase-server failed with no clear reason produced. Perhaps
> > running secondPartTestsExecution in serial will reveal the
> failure/timeout?
>
> That's be sweet.
>

Here's another ticket: HBASE-7350

> Whatever script is handling "Recording test results" could use the default
> > value i mentioned in my initial comment.
>
> Is it the return out of the function inside ./dev-support/test-patch.sh
> called submitJiraComment?
>

I don't think so. The last act of test-patch.sh is to print the banner
"Finished build." and exit. What does Jenkins do after it runs this script?
Regardless, I no longer believe this is causing the endemic build failures.
I'm more suspicious of the xargs/kill.

On Thu, Dec 13, 2012 at 2:33 PM, Nick Dimiduk <[EMAIL PROTECTED]> wrote:

> I don't think so. The last act of test-patch.sh is to print the banner
> "Finished build." and exit. What does Jenkins do after it runs this script?
> Regardless, I no longer believe this is causing the endemic build failures.
> I'm more suspicious of the xargs/kill.
>
Want to try removing it for a while?

St.Ack

On Thu, Dec 13, 2012 at 2:40 PM, Stack <[EMAIL PROTECTED]> wrote:

>
> Want to try removing it for a while?
I attached a couple simple patches to
https://issues.apache.org/jira/browse/HBASE-7350. Maybe someone who knows
more about the different layers of test harness and Jenkins build slave
configuration can comment.

-n

Hi Nick

> Does Jenkins ensure only one HBasePatchProcess runs on a given host at a
time?
Yes, there is no concurrent build on the same machine. You're always alone.
It's unlikely to change: it's already difficult enough to parallelize
things for a single product; doing it between multiple products would be
crazy.

> Tests for hbase-server failed with no clear reason produced. Perhaps
running secondPartTestsExecution in serial will reveal the failure/timeout?
We could add a check at the end. Something like

jps | grep surefirebooter | cut -d ' ' -f 1 | xargs -n 1 jstack | grep Test

I'm going to add this in the patches already proposed in HBASE-7350.

Thanks for looking at this!

N.

On Fri, Dec 14, 2012 at 12:47 AM, Nick Dimiduk <[EMAIL PROTECTED]> wrote:

> On Thu, Dec 13, 2012 at 2:40 PM, Stack <[EMAIL PROTECTED]> wrote:
>
> >
> > Want to try removing it for a while?
>
>
> I attached a couple simple patches to
> https://issues.apache.org/jira/browse/HBASE-7350. Maybe someone who knows
> more about the different layers of test harness and Jenkins build slave
> configuration can comment.
>
> -n
>

Looks like there might be a hanging test:

Tests run: 1238, Failures: 0, Errors: 0, Skipped: 11

Cheers

On Thu, Dec 13, 2012 at 10:24 AM, Nick Dimiduk <[EMAIL PROTECTED]> wrote:

> Hi guys,
>
> Every build in recent history run by the PreCommit job [0] has failed. For
> some of them, this is the usual suspect of a patch not applying cleanly.
> But for many, the issue looks like a simple configuration issue. Case in
> point, this one [1] looks like Jenkins simply failed to read a value from
> the JIRA. Scroll to the very bottom, you'll see what I mean. Is there a way
> to set a default value for this field on the Jenkins side (something like
> empty string, perhaps)?
>
> A CI system is only as good as it is reliable...
>
> Thanks,
> Nick
>
> [0]: https://builds.apache.org/job/PreCommit-HBASE-Build/
> [1]: https://builds.apache.org/job/PreCommit-HBASE-Build/3526/consoleText
>