Hi,
HBase 0.92 upgraded hadoop to 1.0.3 today.
We started seeing some test failures which we didn't see before. e.g.:
https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/410/testReport/org.apache.hadoop.hbase.mapred/TestTableMapReduce/testMultiRegionTable/
Here is the interesting part:

Caused by: java.lang.ClassNotFoundException: org.apache.commons.io.FileUtils
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 15 more

In hadoop 1.0, I see:
commons-io.version=2.1

HBase doesn't specify commons-io version.

When I run the same test on MacBook, I don't see the test fail.

commons-io in local mvn repo is 1.3.2

If someone can give me hint on why the unit test failed, that would be
great.

I believe we upgraded to commons-io 3.0 in 1.0.3. Did you check the 1.0.3 pom?

-Todd

On Thu, May 17, 2012 at 8:01 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> Hi,
> HBase 0.92 upgraded hadoop to 1.0.3 today.
> We started seeing some test failures which we didn't see before. e.g.:
> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/410/testReport/org.apache.hadoop.hbase.mapred/TestTableMapReduce/testMultiRegionTable/
> Here is the interesting part:
>
> Caused by: java.lang.ClassNotFoundException: org.apache.commons.io.FileUtils
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>        ... 15 more
>
> In hadoop 1.0, I see:
> commons-io.version=2.1
>
> HBase doesn't specify commons-io version.
>
> When I run the same test on MacBook, I don't see the test fail.
>
> commons-io in local mvn repo is 1.3.2
>
> If someone can give me hint on why the unit test failed, that would be
> great.

--
Todd Lipcon
Software Engineer, Cloudera

Correction to my first email:
I refreshed my HBase 0.92 workspace and I could see the same test failure.

Under ~/.m2/repository/org/apache/commons/commons-io, I still see 1.3.2

I ran 'mvn dependency:tree' under 0.92 and don't see commons-io in the
output.

Do we need to include commons-io explicitly in our pom.xml ?

For Todd's question:

LM-SJN-00713032:205-hadoop zhihyu$ svn info
Path: .
URL: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1
Repository Root: https://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1339961
Node Kind: directory
Schedule: normal
Last Changed Author: mattf
Last Changed Rev: 1339945
Last Changed Date: 2012-05-17 18:25:57 -0700 (Thu, 17 May 2012)

The line I referenced in my first email came from ivy/libraries.properties

On Thu, May 17, 2012 at 8:06 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> I believe we upgraded to commons-io 3.0 in 1.0.3. Did you check the 1.0.3
> pom?
>
> -Todd
>
> On Thu, May 17, 2012 at 8:01 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > Hi,
> > HBase 0.92 upgraded hadoop to 1.0.3 today.
> > We started seeing some test failures which we didn't see before. e.g.:
> >
> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/410/testReport/org.apache.hadoop.hbase.mapred/TestTableMapReduce/testMultiRegionTable/
> > Here is the interesting part:
> >
> > Caused by: java.lang.ClassNotFoundException:
> org.apache.commons.io.FileUtils
> >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >        at java.security.AccessController.doPrivileged(Native Method)
> >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >        ... 15 more
> >
> > In hadoop 1.0, I see:
> > commons-io.version=2.1
> >
> > HBase doesn't specify commons-io version.
> >
> > When I run the same test on MacBook, I don't see the test fail.
> >
> > commons-io in local mvn repo is 1.3.2
> >
> > If someone can give me hint on why the unit test failed, that would be
> > great.
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

I changed hadoop version to 1.0.2 and the test passed.

I would tend to say 1.0.3 had a regression in this regard.

Cheers

On Thu, May 17, 2012 at 8:11 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> Correction to my first email:
> I refreshed my HBase 0.92 workspace and I could see the same test failure.
>
> Under ~/.m2/repository/org/apache/commons/commons-io, I still see 1.3.2
>
> I ran 'mvn dependency:tree' under 0.92 and don't see commons-io in the
> output.
>
> Do we need to include commons-io explicitly in our pom.xml ?
>
> For Todd's question:
>
> LM-SJN-00713032:205-hadoop zhihyu$ svn info
> Path: .
> URL: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1
> Repository Root: https://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1339961
> Node Kind: directory
> Schedule: normal
> Last Changed Author: mattf
> Last Changed Rev: 1339945
> Last Changed Date: 2012-05-17 18:25:57 -0700 (Thu, 17 May 2012)
>
> The line I referenced in my first email came from ivy/libraries.properties
>
>
> On Thu, May 17, 2012 at 8:06 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> I believe we upgraded to commons-io 3.0 in 1.0.3. Did you check the 1.0.3
>> pom?
>>
>> -Todd
>>
>> On Thu, May 17, 2012 at 8:01 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> > Hi,
>> > HBase 0.92 upgraded hadoop to 1.0.3 today.
>> > We started seeing some test failures which we didn't see before. e.g.:
>> >
>> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/410/testReport/org.apache.hadoop.hbase.mapred/TestTableMapReduce/testMultiRegionTable/
>> > Here is the interesting part:
>> >
>> > Caused by: java.lang.ClassNotFoundException:
>> org.apache.commons.io.FileUtils
>> >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> >        at java.security.AccessController.doPrivileged(Native Method)
>> >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> >        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>> >        ... 15 more
>> >
>> > In hadoop 1.0, I see:
>> > commons-io.version=2.1
>> >
>> > HBase doesn't specify commons-io version.
>> >
>> > When I run the same test on MacBook, I don't see the test fail.
>> >
>> > commons-io in local mvn repo is 1.3.2
>> >
>> > If someone can give me hint on why the unit test failed, that would be
>> > great.
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>

Looks like this is caused by HADOOP-6963, which introduced commons-io
2.1 in ivy.xml but forgot to update the hadoop-core-pom-template.xml.
The old version (1.3.2) of the commons-io is probably an existing
transient dependency. Please file a blocker/critical bug for
1.0.4/1.1.0. The workaround would be explicitly include commons-io 2.1
in downstream projects, but it's not the right thing to do.

__Luke

On Thu, May 17, 2012 at 9:37 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> I changed hadoop version to 1.0.2 and the test passed.
>
> I would tend to say 1.0.3 had a regression in this regard.
>
> Cheers
>
> On Thu, May 17, 2012 at 8:11 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
>> Correction to my first email:
>> I refreshed my HBase 0.92 workspace and I could see the same test failure.
>>
>> Under ~/.m2/repository/org/apache/commons/commons-io, I still see 1.3.2
>>
>> I ran 'mvn dependency:tree' under 0.92 and don't see commons-io in the
>> output.
>>
>> Do we need to include commons-io explicitly in our pom.xml ?
>>
>> For Todd's question:
>>
>> LM-SJN-00713032:205-hadoop zhihyu$ svn info
>> Path: .
>> URL: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1
>> Repository Root: https://svn.apache.org/repos/asf
>> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
>> Revision: 1339961
>> Node Kind: directory
>> Schedule: normal
>> Last Changed Author: mattf
>> Last Changed Rev: 1339945
>> Last Changed Date: 2012-05-17 18:25:57 -0700 (Thu, 17 May 2012)
>>
>> The line I referenced in my first email came from ivy/libraries.properties
>>
>>
>> On Thu, May 17, 2012 at 8:06 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>>
>>> I believe we upgraded to commons-io 3.0 in 1.0.3. Did you check the 1.0.3
>>> pom?
>>>
>>> -Todd
>>>
>>> On Thu, May 17, 2012 at 8:01 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>> > Hi,
>>> > HBase 0.92 upgraded hadoop to 1.0.3 today.
>>> > We started seeing some test failures which we didn't see before. e.g.:
>>> >
>>> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/410/testReport/org.apache.hadoop.hbase.mapred/TestTableMapReduce/testMultiRegionTable/
>>> > Here is the interesting part:
>>> >
>>> > Caused by: java.lang.ClassNotFoundException:
>>> org.apache.commons.io.FileUtils
>>> >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> >        at java.security.AccessController.doPrivileged(Native Method)
>>> >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>> >        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>> >        ... 15 more
>>> >
>>> > In hadoop 1.0, I see:
>>> > commons-io.version=2.1
>>> >
>>> > HBase doesn't specify commons-io version.
>>> >
>>> > When I run the same test on MacBook, I don't see the test fail.
>>> >
>>> > commons-io in local mvn repo is 1.3.2
>>> >
>>> > If someone can give me hint on why the unit test failed, that would be
>>> > great.
>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>>
>>

HADOOP-8417 has been filed.

On Sat, May 19, 2012 at 5:59 PM, Luke Lu <[EMAIL PROTECTED]> wrote:

> Looks like this is caused by HADOOP-6963, which introduced commons-io
> 2.1 in ivy.xml but forgot to update the hadoop-core-pom-template.xml.
> The old version (1.3.2) of the commons-io is probably an existing
> transient dependency. Please file a blocker/critical bug for
> 1.0.4/1.1.0. The workaround would be explicitly include commons-io 2.1
> in downstream projects, but it's not the right thing to do.
>
> __Luke
>
> On Thu, May 17, 2012 at 9:37 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > I changed hadoop version to 1.0.2 and the test passed.
> >
> > I would tend to say 1.0.3 had a regression in this regard.
> >
> > Cheers
> >
> > On Thu, May 17, 2012 at 8:11 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> >> Correction to my first email:
> >> I refreshed my HBase 0.92 workspace and I could see the same test
> failure.
> >>
> >> Under ~/.m2/repository/org/apache/commons/commons-io, I still see 1.3.2
> >>
> >> I ran 'mvn dependency:tree' under 0.92 and don't see commons-io in the
> >> output.
> >>
> >> Do we need to include commons-io explicitly in our pom.xml ?
> >>
> >> For Todd's question:
> >>
> >> LM-SJN-00713032:205-hadoop zhihyu$ svn info
> >> Path: .
> >> URL: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1
> >> Repository Root: https://svn.apache.org/repos/asf
> >> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> >> Revision: 1339961
> >> Node Kind: directory
> >> Schedule: normal
> >> Last Changed Author: mattf
> >> Last Changed Rev: 1339945
> >> Last Changed Date: 2012-05-17 18:25:57 -0700 (Thu, 17 May 2012)
> >>
> >> The line I referenced in my first email came from
> ivy/libraries.properties
> >>
> >>
> >> On Thu, May 17, 2012 at 8:06 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> >>
> >>> I believe we upgraded to commons-io 3.0 in 1.0.3. Did you check the
> 1.0.3
> >>> pom?
> >>>
> >>> -Todd
> >>>
> >>> On Thu, May 17, 2012 at 8:01 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >>> > Hi,
> >>> > HBase 0.92 upgraded hadoop to 1.0.3 today.
> >>> > We started seeing some test failures which we didn't see before.
> e.g.:
> >>> >
> >>>
> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/410/testReport/org.apache.hadoop.hbase.mapred/TestTableMapReduce/testMultiRegionTable/
> >>> > Here is the interesting part:
> >>> >
> >>> > Caused by: java.lang.ClassNotFoundException:
> >>> org.apache.commons.io.FileUtils
> >>> >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>> >        at java.security.AccessController.doPrivileged(Native Method)
> >>> >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>> >        at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >>> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>> >        ... 15 more
> >>> >
> >>> > In hadoop 1.0, I see:
> >>> > commons-io.version=2.1
> >>> >
> >>> > HBase doesn't specify commons-io version.
> >>> >
> >>> > When I run the same test on MacBook, I don't see the test fail.
> >>> >
> >>> > commons-io in local mvn repo is 1.3.2
> >>> >
> >>> > If someone can give me hint on why the unit test failed, that would
> be
> >>> > great.
> >>>
> >>>
> >>>
> >>> --
> >>> Todd Lipcon
> >>> Software Engineer, Cloudera
> >>>
> >>
> >>
>

I don't see 1.0.4 in Fix Versions dropdown list.

Personally this defect should at least be documented in 1.0.3 release
notes. Users would have to package commons-io 2.1 themselves so that
map/reduce jobs can succeed.

On Sat, May 19, 2012 at 6:25 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> HADOOP-8417 has been filed.
>
>
> On Sat, May 19, 2012 at 5:59 PM, Luke Lu <[EMAIL PROTECTED]> wrote:
>
>> Looks like this is caused by HADOOP-6963, which introduced commons-io
>> 2.1 in ivy.xml but forgot to update the hadoop-core-pom-template.xml.
>> The old version (1.3.2) of the commons-io is probably an existing
>> transient dependency. Please file a blocker/critical bug for
>> 1.0.4/1.1.0. The workaround would be explicitly include commons-io 2.1
>> in downstream projects, but it's not the right thing to do.
>>
>> __Luke
>>
>> On Thu, May 17, 2012 at 9:37 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> > I changed hadoop version to 1.0.2 and the test passed.
>> >
>> > I would tend to say 1.0.3 had a regression in this regard.
>> >
>> > Cheers
>> >
>> > On Thu, May 17, 2012 at 8:11 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> >
>> >> Correction to my first email:
>> >> I refreshed my HBase 0.92 workspace and I could see the same test
>> failure.
>> >>
>> >> Under ~/.m2/repository/org/apache/commons/commons-io, I still see 1.3.2
>> >>
>> >> I ran 'mvn dependency:tree' under 0.92 and don't see commons-io in the
>> >> output.
>> >>
>> >> Do we need to include commons-io explicitly in our pom.xml ?
>> >>
>> >> For Todd's question:
>> >>
>> >> LM-SJN-00713032:205-hadoop zhihyu$ svn info
>> >> Path: .
>> >> URL: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1
>> >> Repository Root: https://svn.apache.org/repos/asf
>> >> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
>> >> Revision: 1339961
>> >> Node Kind: directory
>> >> Schedule: normal
>> >> Last Changed Author: mattf
>> >> Last Changed Rev: 1339945
>> >> Last Changed Date: 2012-05-17 18:25:57 -0700 (Thu, 17 May 2012)
>> >>
>> >> The line I referenced in my first email came from
>> ivy/libraries.properties
>> >>
>> >>
>> >> On Thu, May 17, 2012 at 8:06 PM, Todd Lipcon <[EMAIL PROTECTED]>
>> wrote:
>> >>
>> >>> I believe we upgraded to commons-io 3.0 in 1.0.3. Did you check the
>> 1.0.3
>> >>> pom?
>> >>>
>> >>> -Todd
>> >>>
>> >>> On Thu, May 17, 2012 at 8:01 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> >>> > Hi,
>> >>> > HBase 0.92 upgraded hadoop to 1.0.3 today.
>> >>> > We started seeing some test failures which we didn't see before.
>> e.g.:
>> >>> >
>> >>>
>> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/410/testReport/org.apache.hadoop.hbase.mapred/TestTableMapReduce/testMultiRegionTable/
>> >>> > Here is the interesting part:
>> >>> >
>> >>> > Caused by: java.lang.ClassNotFoundException:
>> >>> org.apache.commons.io.FileUtils
>> >>> >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> >>> >        at java.security.AccessController.doPrivileged(Native Method)
>> >>> >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> >>> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> >>> >        at
>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> >>> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>> >>> >        ... 15 more
>> >>> >
>> >>> > In hadoop 1.0, I see:
>> >>> > commons-io.version=2.1
>> >>> >
>> >>> > HBase doesn't specify commons-io version.
>> >>> >
>> >>> > When I run the same test on MacBook, I don't see the test fail.
>> >>> >
>> >>> > commons-io in local mvn repo is 1.3.2
>> >>> >
>> >>> > If someone can give me hint on why the unit test failed, that would
>> be
>> >>> > great.
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Todd Lipcon
>> >>> Software Engineer, Cloudera
>> >>>
>> >>
>> >>
>>
>
>