I upgrade to accumulo-1.4.0 and updated my map/reduce jobs and now they
don't run.  The parent class path has the accumulo-core-1.4.0.jar file
included.  Do the accumulo jar files have to be manually put on a
distribute cache?  Any help is appreciated.

[hadoop@redhat-cloudbase1 placemarks]$ ./runPlacemarkIngester.sh
Found 5 items
drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:13 /accumulo
drwxrwxr-x   - hadoop hadoop          0 2012-05-21 15:06 /data
drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58 /input
drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58 /output
drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:34 /usr
Deleted hdfs://redhat-cloudbase1:9000/output
Deleted hdfs://redhat-cloudbase1:9000/input
12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: Invoking
ToolRunner.run
12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: zooKeeper is
redhat-cloudbase1:2181
12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: instanceName is
NCCT-Cloudbase
12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: timeTableName is
NCCTServicesTimes
12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: geoTableName is
NCCTServicesGeos
12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: metadataTableName
is NCCTServicesMetadata
12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: edgeTableName is
NCCTEdgeTable
12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: userName is
NCCT.Services.Client
12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: password is
*********
12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: visibility is
public,BD2,UNCLASSIFIED-NO_COMPARTMENT-UNRESTRICTED
12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: inputDir is /input
12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: outputDir is
/output
12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48
GMT
12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
environment:host.name=redhat-cloudbase1
12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
environment:java.version=1.6.0_32
12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
environment:java.vendor=Sun Microsystems Inc.
12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
environment:java.home=/usr/java/jdk1.6.0_32/jre
12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
environment:java.class.path=/cloudbase/hadoop-0.20.2/bin/../conf:/usr/ja
va/jdk1.6.0_32/lib/tools.jar:/cloudbase/hadoop-0.20.2/bin/..:/cloudbase/
hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar:/cloudbase/hadoop-0.20.2/bin
/../lib/commons-cli-1.2.jar:/cloudbase/hadoop-0.20.2/bin/../lib/commons-
codec-1.3.jar:/cloudbase/hadoop-0.20.2/bin/../lib/commons-el-1.0.jar:/cl
oudbase/hadoop-0.20.2/bin/../lib/commons-httpclient-3.0.1.jar:/cloudbase
/hadoop-0.20.2/bin/../lib/commons-logging-1.0.4.jar:/cloudbase/hadoop-0.
20.2/bin/../lib/commons-logging-api-1.0.4.jar:/cloudbase/hadoop-0.20.2/b
in/../lib/commons-net-1.4.1.jar:/cloudbase/hadoop-0.20.2/bin/../lib/core
-3.1.1.jar:/cloudbase/hadoop-0.20.2/bin/../lib/hadoop-0.20.2-ant.jar:/cl
oudbase/hadoop-0.20.2/bin/../lib/hadoop-0.20.2-core.jar:/cloudbase/hadoo
p-0.20.2/bin/../lib/hadoop-0.20.2-examples.jar:/cloudbase/hadoop-0.20.2/
bin/../lib/hadoop-0.20.2-test.jar:/cloudbase/hadoop-0.20.2/bin/../lib/ha
doop-0.20.2-tools.jar:/cloudbase/hadoop-0.20.2/bin/../lib/hsqldb-1.8.0.1
0.jar:/cloudbase/hadoop-0.20.2/bin/../lib/jasper-compiler-5.5.12.jar:/cl
oudbase/hadoop-0.20.2/bin/../lib/jasper-runtime-5.5.12.jar:/cloudbase/ha
doop-0.20.2/bin/../lib/jets3t-0.6.1.jar:/cloudbase/hadoop-0.20.2/bin/../
lib/jetty-6.1.14.jar:/cloudbase/hadoop-0.20.2/bin/../lib/jetty-util-6.1.
14.jar:/cloudbase/hadoop-0.20.2/bin/../lib/junit-3.8.1.jar:/cloudbase/ha
doop-0.20.2/bin/../lib/kfs-0.2.2.jar:/cloudbase/hadoop-0.20.2/bin/../lib
/log4j-1.2.15.jar:/cloudbase/hadoop-0.20.2/bin/../lib/mockito-all-1.8.0.
jar:/cloudbase/hadoop-0.20.2/bin/../lib/oro-2.0.8.jar:/cloudbase/hadoop-
0.20.2/bin/../lib/servlet-api-2.5-6.1.14.jar:/cloudbase/hadoop-0.20.2/bi
n/../lib/slf4j-api-1.4.3.jar:/cloudbase/hadoop-0.20.2/bin/../lib/slf4j-l
og4j12-1.4.3.jar:/cloudbase/hadoop-0.20.2/bin/../lib/xmlenc-0.52.jar:/cl
oudbase/hadoop-0.20.2/bin/../lib/jsp-2.1/jsp-2.1.jar:/cloudbase/hadoop-0
.20.2/bin/../lib/jsp-2.1/jsp-api-2.1.jar:/cloudbase/accumulo-1.4.0/lib/l
ibthrift-0.6.1.jar:/cloudbase/accumulo-1.4.0/lib/accumulo-core-1.4.0.jar
0/lib/cloudtrace-1.4.0.jar:/usr/lib/ncct/kxml2-2.3.0.jar:/usr/lib/ncct/x
mlpull-1.1.3.1.jar:/usr/lib/ncct/xstream-1.4.1.jar:/cloudbase/accumulo-1
.4.0/lib/commons-collections-3.2.jar:/cloudbase/accumulo-1.4.0/lib/commo
ns-configuration-1.5.jar:/cloudbase/accumulo-1.4.0/lib/commons-io-1.4.ja
r:/cloudbase/accumulo-1.4.0/lib/commons-jci-core-1.0.jar:/cloudbase/accu
mulo-1.4.0/lib/commons-jci-fam-1.0.jar:/cloudbase/accumulo-1.4.0/lib/com
mons-lang-2.4.jar:/cloudbase/accumulo-1.4.0/lib/commons-logging-1.0.4.ja
r:/cloudbase/accumulo-1.4.0/lib/commons-logging-api-1.0.4.jar:
12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
environment:java.library.path=/cloudbase/hadoop-0.20.2/bin/../lib/native
/Linux-amd64-64
12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
environment:os.name=Linux
12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
environment:os.arch=amd64
12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-131.0.15.el6.x86_64
12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
environment:user.name=hadoop
12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
environment:user.home=/home/hadoop
12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
environment:user.dir=/mnt/hgfs/CSI.Cloudbase/Java/CloudbaseServices/Clou
dbaseIngesters/src/com/comcept/cloudbase/ingesters/placemarks
12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Initiating client
connection, connectString=redhat-cloudbase1:2181 sessionTimeout=30000
watcher=org.apac

If you look at Job.xml, do you see accumulo-core there?  There should
be a link to this file on the jobs page on the tasktracker.

On Tue, May 22, 2012 at 10:40 AM,  <[EMAIL PROTECTED]> wrote:
> I upgrade to accumulo-1.4.0 and updated my map/reduce jobs and now they
> don't run.  The parent class path has the accumulo-core-1.4.0.jar file
> included.  Do the accumulo jar files have to be manually put on a
> distribute cache?  Any help is appreciated.
>
> [hadoop@redhat-cloudbase1 placemarks]$ ./runPlacemarkIngester.sh
> Found 5 items
> drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:13 /accumulo
> drwxrwxr-x   - hadoop hadoop          0 2012-05-21 15:06 /data
> drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58 /input
> drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58 /output
> drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:34 /usr
> Deleted hdfs://redhat-cloudbase1:9000/output
> Deleted hdfs://redhat-cloudbase1:9000/input
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: Invoking
> ToolRunner.run
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: zooKeeper is
> redhat-cloudbase1:2181
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: instanceName is
> NCCT-Cloudbase
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: timeTableName is
> NCCTServicesTimes
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: geoTableName is
> NCCTServicesGeos
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: metadataTableName
> is NCCTServicesMetadata
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: edgeTableName is
> NCCTEdgeTable
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: userName is
> NCCT.Services.Client
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: password is
> *********
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: visibility is
> public,BD2,UNCLASSIFIED-NO_COMPARTMENT-UNRESTRICTED
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: inputDir is /input
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: outputDir is
> /output
> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48
> GMT
> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
> environment:host.name=redhat-cloudbase1
> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
> environment:java.version=1.6.0_32
> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
> environment:java.vendor=Sun Microsystems Inc.
> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
> environment:java.home=/usr/java/jdk1.6.0_32/jre
> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
> environment:java.class.path=/cloudbase/hadoop-0.20.2/bin/../conf:/usr/ja
> va/jdk1.6.0_32/lib/tools.jar:/cloudbase/hadoop-0.20.2/bin/..:/cloudbase/
> hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar:/cloudbase/hadoop-0.20.2/bin
> /../lib/commons-cli-1.2.jar:/cloudbase/hadoop-0.20.2/bin/../lib/commons-
> codec-1.3.jar:/cloudbase/hadoop-0.20.2/bin/../lib/commons-el-1.0.jar:/cl
> oudbase/hadoop-0.20.2/bin/../lib/commons-httpclient-3.0.1.jar:/cloudbase
> /hadoop-0.20.2/bin/../lib/commons-logging-1.0.4.jar:/cloudbase/hadoop-0.
> 20.2/bin/../lib/commons-logging-api-1.0.4.jar:/cloudbase/hadoop-0.20.2/b
> in/../lib/commons-net-1.4.1.jar:/cloudbase/hadoop-0.20.2/bin/../lib/core
> -3.1.1.jar:/cloudbase/hadoop-0.20.2/bin/../lib/hadoop-0.20.2-ant.jar:/cl
> oudbase/hadoop-0.20.2/bin/../lib/hadoop-0.20.2-core.jar:/cloudbase/hadoo
> p-0.20.2/bin/../lib/hadoop-0.20.2-examples.jar:/cloudbase/hadoop-0.20.2/
> bin/../lib/hadoop-0.20.2-test.jar:/cloudbase/hadoop-0.20.2/bin/../lib/ha
> doop-0.20.2-tools.jar:/cloudbase/hadoop-0.20.2/bin/../lib/hsqldb-1.8.0.1
> 0.jar:/cloudbase/hadoop-0.20.2/bin/../lib/jasper-compiler-5.5.12.jar:/cl
> oudbase/hadoop-0.20.2/bin/../lib/jasper-runtime-5.5.12.jar:/cloudbase/ha
> doop-0.20.2/bin/../lib/jets3t-0.6.1.jar:/cloudbase/hadoop-0.20.2/bin/../
> lib/jetty-6.1.14.jar:/cloudbase/hadoop-0.20.2/bin/../lib/jetty-util-6.1.
> 14.jar:/cloudbase/hadoop-0.20.2/bin/../lib/junit-3.8.1.jar:/cloudbase/ha

No, it's not in the job.xml file.  A reference to AccumuloFileOutputFormat is but not the accumulo-core-1.4.0.jar.  The Job*.jar file is referenced there.  I was under the impression that the child jvm inherits the parent jvm attributes which makes the job*.jar file.  Is that not correct?

-----Original Message-----
From: Keith Turner [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, May 22, 2012 12:07
To: [EMAIL PROTECTED]
Subject: Re: AcculumoFileOutputFormat class cannot be found by child jvm

If you look at Job.xml, do you see accumulo-core there?  There should be a link to this file on the jobs page on the tasktracker.

On Tue, May 22, 2012 at 10:40 AM,  <[EMAIL PROTECTED]> wrote:
> I upgrade to accumulo-1.4.0 and updated my map/reduce jobs and now
> they don't run.  The parent class path has the accumulo-core-1.4.0.jar
> file included.  Do the accumulo jar files have to be manually put on a
> distribute cache?  Any help is appreciated.
>
> [hadoop@redhat-cloudbase1 placemarks]$ ./runPlacemarkIngester.sh Found
> 5 items drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:13
> /accumulo drwxrwxr-x   - hadoop hadoop          0 2012-05-21 15:06
> /data drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58 /input
> drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58 /output
> drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:34 /usr Deleted
> hdfs://redhat-cloudbase1:9000/output
> Deleted hdfs://redhat-cloudbase1:9000/input
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: Invoking
> ToolRunner.run
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: zooKeeper is
> redhat-cloudbase1:2181
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: instanceName is
> NCCT-Cloudbase
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: timeTableName is
> NCCTServicesTimes
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: geoTableName is
> NCCTServicesGeos
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: metadataTableName
> is NCCTServicesMetadata
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: edgeTableName is
> NCCTEdgeTable
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: userName is
> NCCT.Services.Client
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: password is
> *********
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: visibility is
> public,BD2,UNCLASSIFIED-NO_COMPARTMENT-UNRESTRICTED
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: inputDir is
> /input
> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: outputDir is
> /output
> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48
> GMT
> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
> environment:host.name=redhat-cloudbase1
> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
> environment:java.version=1.6.0_32
> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
> environment:java.vendor=Sun Microsystems Inc.
> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
> environment:java.home=/usr/java/jdk1.6.0_32/jre
> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
> environment:java.class.path=/cloudbase/hadoop-0.20.2/bin/../conf:/usr/
> ja
> va/jdk1.6.0_32/lib/tools.jar:/cloudbase/hadoop-0.20.2/bin/..:/cloudbas
> e/
> hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar:/cloudbase/hadoop-0.20.2/b
> in
> /../lib/commons-cli-1.2.jar:/cloudbase/hadoop-0.20.2/bin/../lib/common
> s-
> codec-1.3.jar:/cloudbase/hadoop-0.20.2/bin/../lib/commons-el-1.0.jar:/
> cl
> oudbase/hadoop-0.20.2/bin/../lib/commons-httpclient-3.0.1.jar:/cloudba
> se
> /hadoop-0.20.2/bin/../lib/commons-logging-1.0.4.jar:/cloudbase/hadoop-0.
> 20.2/bin/../lib/commons-logging-api-1.0.4.jar:/cloudbase/hadoop-0.20.2
> /b
> in/../lib/commons-net-1.4.1.jar:/cloudbase/hadoop-0.20.2/bin/../lib/co
> re
> -3.1.1.jar:/cloudbase/hadoop-0.20.2/bin/../lib/hadoop-0.20.2-ant.jar:/
> cl
> oudbase/hadoop-0.20.2/bin/../lib/hadoop-0.20.2-core.jar:/cloudbase/had
> oo

I think it should be in the job.xml file.  The way the class path for
the java program that starts map reduce is configured is different
from how the classpath for the remote task are configured.   So your
program that starts the map reduce job has AccumuloFileOutputFormat on
the class path because you use a static method on it and it runs.  So
I am thinking its not in --libjars for some reason or maybe you are
not using Hadoop Tool?

On Tue, May 22, 2012 at 1:54 PM,  <[EMAIL PROTECTED]> wrote:
> No, it's not in the job.xml file.  A reference to AccumuloFileOutputFormat is but not the accumulo-core-1.4.0.jar.  The Job*.jar file is referenced there.  I was under the impression that the child jvm inherits the parent jvm attributes which makes the job*.jar file.  Is that not correct?
>
> -----Original Message-----
> From: Keith Turner [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, May 22, 2012 12:07
> To: [EMAIL PROTECTED]
> Subject: Re: AcculumoFileOutputFormat class cannot be found by child jvm
>
> If you look at Job.xml, do you see accumulo-core there?  There should be a link to this file on the jobs page on the tasktracker.
>
> On Tue, May 22, 2012 at 10:40 AM,  <[EMAIL PROTECTED]> wrote:
>> I upgrade to accumulo-1.4.0 and updated my map/reduce jobs and now
>> they don't run.  The parent class path has the accumulo-core-1.4.0.jar
>> file included.  Do the accumulo jar files have to be manually put on a
>> distribute cache?  Any help is appreciated.
>>
>> [hadoop@redhat-cloudbase1 placemarks]$ ./runPlacemarkIngester.sh Found
>> 5 items drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:13
>> /accumulo drwxrwxr-x   - hadoop hadoop          0 2012-05-21 15:06
>> /data drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58 /input
>> drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58 /output
>> drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:34 /usr Deleted
>> hdfs://redhat-cloudbase1:9000/output
>> Deleted hdfs://redhat-cloudbase1:9000/input
>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: Invoking
>> ToolRunner.run
>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: zooKeeper is
>> redhat-cloudbase1:2181
>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: instanceName is
>> NCCT-Cloudbase
>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: timeTableName is
>> NCCTServicesTimes
>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: geoTableName is
>> NCCTServicesGeos
>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: metadataTableName
>> is NCCTServicesMetadata
>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: edgeTableName is
>> NCCTEdgeTable
>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: userName is
>> NCCT.Services.Client
>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: password is
>> *********
>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: visibility is
>> public,BD2,UNCLASSIFIED-NO_COMPARTMENT-UNRESTRICTED
>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: inputDir is
>> /input
>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: outputDir is
>> /output
>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>> environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48
>> GMT
>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>> environment:host.name=redhat-cloudbase1
>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>> environment:java.version=1.6.0_32
>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>> environment:java.vendor=Sun Microsystems Inc.
>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>> environment:java.home=/usr/java/jdk1.6.0_32/jre
>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>> environment:java.class.path=/cloudbase/hadoop-0.20.2/bin/../conf:/usr/
>> ja
>> va/jdk1.6.0_32/lib/tools.jar:/cloudbase/hadoop-0.20.2/bin/..:/cloudbas
>> e/
>> hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar:/cloudbase/hadoop-0.20.2/b
>> in
>> /../lib/commons-cli-1.2.jar:/cloudbase/hadoop-0.20.2/bin/../lib/common

Make sure that the paths you pass to `-libjars` are correct. I have
noticed behavior with Hadoop where it will silently drop the end of a
libjars string if the paths do not exist.

On Tue, May 22, 2012 at 11:03 AM, Keith Turner <[EMAIL PROTECTED]> wrote:
> I think it should be in the job.xml file.  The way the class path for
> the java program that starts map reduce is configured is different
> from how the classpath for the remote task are configured.   So your
> program that starts the map reduce job has AccumuloFileOutputFormat on
> the class path because you use a static method on it and it runs.  So
> I am thinking its not in --libjars for some reason or maybe you are
> not using Hadoop Tool?
>
> On Tue, May 22, 2012 at 1:54 PM,  <[EMAIL PROTECTED]> wrote:
>> No, it's not in the job.xml file.  A reference to AccumuloFileOutputFormat is but not the accumulo-core-1.4.0.jar.  The Job*.jar file is referenced there.  I was under the impression that the child jvm inherits the parent jvm attributes which makes the job*.jar file.  Is that not correct?
>>
>> -----Original Message-----
>> From: Keith Turner [mailto:[EMAIL PROTECTED]]
>> Sent: Tuesday, May 22, 2012 12:07
>> To: [EMAIL PROTECTED]
>> Subject: Re: AcculumoFileOutputFormat class cannot be found by child jvm
>>
>> If you look at Job.xml, do you see accumulo-core there?  There should be a link to this file on the jobs page on the tasktracker.
>>
>> On Tue, May 22, 2012 at 10:40 AM,  <[EMAIL PROTECTED]> wrote:
>>> I upgrade to accumulo-1.4.0 and updated my map/reduce jobs and now
>>> they don't run.  The parent class path has the accumulo-core-1.4.0.jar
>>> file included.  Do the accumulo jar files have to be manually put on a
>>> distribute cache?  Any help is appreciated.
>>>
>>> [hadoop@redhat-cloudbase1 placemarks]$ ./runPlacemarkIngester.sh Found
>>> 5 items drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:13
>>> /accumulo drwxrwxr-x   - hadoop hadoop          0 2012-05-21 15:06
>>> /data drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58 /input
>>> drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58 /output
>>> drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:34 /usr Deleted
>>> hdfs://redhat-cloudbase1:9000/output
>>> Deleted hdfs://redhat-cloudbase1:9000/input
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: Invoking
>>> ToolRunner.run
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: zooKeeper is
>>> redhat-cloudbase1:2181
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: instanceName is
>>> NCCT-Cloudbase
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: timeTableName is
>>> NCCTServicesTimes
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: geoTableName is
>>> NCCTServicesGeos
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: metadataTableName
>>> is NCCTServicesMetadata
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: edgeTableName is
>>> NCCTEdgeTable
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: userName is
>>> NCCT.Services.Client
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: password is
>>> *********
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: visibility is
>>> public,BD2,UNCLASSIFIED-NO_COMPARTMENT-UNRESTRICTED
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: inputDir is
>>> /input
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: outputDir is
>>> /output
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>> environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48
>>> GMT
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>> environment:host.name=redhat-cloudbase1
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>> environment:java.version=1.6.0_32
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>> environment:java.vendor=Sun Microsystems Inc.
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>> environment:java.home=/usr/java/jdk1.6.0_32/jre
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client

On 05/22/2012 10:40 AM, [EMAIL PROTECTED] wrote:
> I upgrade to accumulo-1.4.0 and updated my map/reduce jobs and now they
> don't run.  The parent class path has the accumulo-core-1.4.0.jar file
> included.  Do the accumulo jar files have to be manually put on a
> distribute cache?  Any help is appreciated.

Just to check: did you replace the Accumulo JAR files on all the cluster
nodes?

Right now I'm using stand-alone mode, but is there another place I need
to put the jar file?

-----Original Message-----
From: John Armstrong [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, May 22, 2012 09:49
To: [EMAIL PROTECTED]
Subject: Re: AcculumoFileOutputFormat class cannot be found by child jvm

On 05/22/2012 10:40 AM, [EMAIL PROTECTED] wrote:
> I upgrade to accumulo-1.4.0 and updated my map/reduce jobs and now
> they don't run.  The parent class path has the accumulo-core-1.4.0.jar

> file included.  Do the accumulo jar files have to be manually put on a

> distribute cache?  Any help is appreciated.

Just to check: did you replace the Accumulo JAR files on all the cluster
nodes?

On 05/22/2012 10:52 AM, [EMAIL PROTECTED] wrote:
> Right now I'm using stand-alone mode, but is there another place I need
> to put the jar file?

I'm not sure; I haven't really used stand-alone mode.  Hopefully someone
who has will know where its child JVMs get their classpaths from.

Does your script utilize $ACCUMULO_HOME/bin/tool.sh to kick off the
mapreduce? That script is similar to hadoop jar, but it will libjar the
accumulo libraries for you.

John

On Tue, May 22, 2012 at 10:55 AM, <[EMAIL PROTECTED]> wrote:

> Right now I'm using stand-alone mode, but is there another place I need
> to put the jar file?
>
> -----Original Message-----
> From: John Armstrong [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, May 22, 2012 09:49
> To: [EMAIL PROTECTED]
> Subject: Re: AcculumoFileOutputFormat class cannot be found by child jvm
>
> On 05/22/2012 10:40 AM, [EMAIL PROTECTED] wrote:
> > I upgrade to accumulo-1.4.0 and updated my map/reduce jobs and now
> > they don't run.  The parent class path has the accumulo-core-1.4.0.jar
>
> > file included.  Do the accumulo jar files have to be manually put on a
>
> > distribute cache?  Any help is appreciated.
>
> Just to check: did you replace the Accumulo JAR files on all the cluster
> nodes?
>
>

Yep.  Here's the script I'm using...everything is happy until the job
executes under the configuration that uses AccumuloFileOutputFormat
class...

 

HADOOP_BIN=/cloudbase/hadoop-0.20.2/bin

ACCUMULO_BIN=/cloudbase/accumulo-1.4.0/bin

 

INGESTER_JAR=/mnt/hgfs/CSI.Cloudbase/Java/CloudbaseServices/out/artifact
s/CloudbaseIngesters/CloudbaseIngesters.jar

PLACEMARK_CLASS=com.comcept.cloudbase.ingesters.placemarks.PlacemarkInge
ster

CONFIG=/mnt/hgfs/CSI.Cloudbase/Java/CloudbaseServices/out/artifacts/Clou
dbaseIngesters/placemark-config.xml

 

KXML_JAR=/usr/lib/ncct/kxml2-2.3.0.jar

XMLPULL_JAR=/usr/lib/ncct/xmlpull-1.1.3.1.jar

XSTREAM_JAR=/usr/lib/ncct/xstream-1.4.1.jar

 

INGESTER_LIBS=$KXML_JAR,$XMLPULL_JAR,$XSTREAM_JAR

 

$HADOOP_BIN/hadoop dfs -ls /

$HADOOP_BIN/hadoop dfs -rmr /output

$HADOOP_BIN/hadoop dfs -rmr /input

$HADOOP_BIN/hadoop dfs -mkdir /input

$HADOOP_BIN/hadoop dfs -mkdir /output

$HADOOP_BIN/hadoop dfs -mkdir /output/pfailures

$HADOOP_BIN/hadoop dfs -mkdir /output/gfailures

$HADOOP_BIN/hadoop dfs -mkdir /output/efailures

$HADOOP_BIN/hadoop dfs -mkdir /output/tfailures

$HADOOP_BIN/hadoop dfs -put ./*.kml /input

 

$ACCUMULO_BIN/tool.sh $INGESTER_JAR $PLACEMARK_CLASS -libjars
$INGESTER_LIBS -c $CONFIG

 

 

Here is the code that initializes the first job in the chain...

 

                conf.set(_sVisTag, ic.getVisibility());

 

Job job = new Job(conf, "NCCT Placemark Ingester");

job.setJarByClass(this.getClass());

                job.setInputFormatClass(TextInputFormat.class);

                job.setMapperClass(PlacemarkMapClass.class);

                job.setMapOutputKeyClass(Text.class);

                job.setMapOutputValueClass(Text.class);

                job.setReducerClass(PlacemarkReduceClass.class);

 
job.setOutputFormatClass(AccumuloFileOutputFormat.class);

 

AccumuloFileOutputFormat.setZooKeeperInstance(conf, ic.getInstance(),
ic.getZooKeeper());

                Instance instance = new
ZooKeeperInstance(ic.getInstance(), ic.getZooKeeper());

                Connector connector instance.getConnector(ic.getUserName(), password);

                TextInputFormat.setInputPaths(job,new
Path(ic.getHdfsInput()));

                AccumuloFileOutputFormat.setOutputPath(job, new
Path(ic.getHdfsOutput() + "/pfiles"));

 

                job.waitForCompletion(true);

 
connector.tableOperations().importDirectory(ic.getMetaTable(),
ic.getHdfsOutput() + "/pfiles", ic.getHdfsOutput() + "/pfailures",
false);

 

From: John Vines [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, May 22, 2012 09:57
To: [EMAIL PROTECTED]
Subject: Re: AcculumoFileOutputFormat class cannot be found by child jvm

 

Does your script utilize $ACCUMULO_HOME/bin/tool.sh to kick off the
mapreduce? That script is similar to hadoop jar, but it will libjar the
accumulo libraries for you.

John

On Tue, May 22, 2012 at 10:55 AM, <[EMAIL PROTECTED]> wrote:

Right now I'm using stand-alone mode, but is there another place I need
to put the jar file?
-----Original Message-----
From: John Armstrong [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, May 22, 2012 09:49
To: [EMAIL PROTECTED]
Subject: Re: AcculumoFileOutputFormat class cannot be found by child jvm

On 05/22/2012 10:40 AM, [EMAIL PROTECTED] wrote:
> I upgrade to accumulo-1.4.0 and updated my map/reduce jobs and now
> they don't run.  The parent class path has the accumulo-core-1.4.0.jar

> file included.  Do the accumulo jar files have to be manually put on a

> distribute cache?  Any help is appreciated.

Just to check: did you replace the Accumulo JAR files on all the cluster
nodes?

 

I'm wondering if our tool.sh plays nice with an explicit -libjars.

John

On Tue, May 22, 2012 at 2:07 PM, William Slacum <[EMAIL PROTECTED]> wrote:

> Make sure that the paths you pass to `-libjars` are correct. I have
> noticed behavior with Hadoop where it will silently drop the end of a
> libjars string if the paths do not exist.
>
> On Tue, May 22, 2012 at 11:03 AM, Keith Turner <[EMAIL PROTECTED]> wrote:
> > I think it should be in the job.xml file.  The way the class path for
> > the java program that starts map reduce is configured is different
> > from how the classpath for the remote task are configured.   So your
> > program that starts the map reduce job has AccumuloFileOutputFormat on
> > the class path because you use a static method on it and it runs.  So
> > I am thinking its not in --libjars for some reason or maybe you are
> > not using Hadoop Tool?
> >
> > On Tue, May 22, 2012 at 1:54 PM,  <[EMAIL PROTECTED]> wrote:
> >> No, it's not in the job.xml file.  A reference to
> AccumuloFileOutputFormat is but not the accumulo-core-1.4.0.jar.  The
> Job*.jar file is referenced there.  I was under the impression that the
> child jvm inherits the parent jvm attributes which makes the job*.jar file.
>  Is that not correct?
> >>
> >> -----Original Message-----
> >> From: Keith Turner [mailto:[EMAIL PROTECTED]]
> >> Sent: Tuesday, May 22, 2012 12:07
> >> To: [EMAIL PROTECTED]
> >> Subject: Re: AcculumoFileOutputFormat class cannot be found by child jvm
> >>
> >> If you look at Job.xml, do you see accumulo-core there?  There should
> be a link to this file on the jobs page on the tasktracker.
> >>
> >> On Tue, May 22, 2012 at 10:40 AM,  <[EMAIL PROTECTED]> wrote:
> >>> I upgrade to accumulo-1.4.0 and updated my map/reduce jobs and now
> >>> they don't run.  The parent class path has the accumulo-core-1.4.0.jar
> >>> file included.  Do the accumulo jar files have to be manually put on a
> >>> distribute cache?  Any help is appreciated.
> >>>
> >>> [hadoop@redhat-cloudbase1 placemarks]$ ./runPlacemarkIngester.sh Found
> >>> 5 items drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:13
> >>> /accumulo drwxrwxr-x   - hadoop hadoop          0 2012-05-21 15:06
> >>> /data drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58 /input
> >>> drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58 /output
> >>> drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:34 /usr Deleted
> >>> hdfs://redhat-cloudbase1:9000/output
> >>> Deleted hdfs://redhat-cloudbase1:9000/input
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: Invoking
> >>> ToolRunner.run
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: zooKeeper is
> >>> redhat-cloudbase1:2181
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: instanceName is
> >>> NCCT-Cloudbase
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: timeTableName is
> >>> NCCTServicesTimes
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: geoTableName is
> >>> NCCTServicesGeos
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: metadataTableName
> >>> is NCCTServicesMetadata
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: edgeTableName is
> >>> NCCTEdgeTable
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: userName is
> >>> NCCT.Services.Client
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: password is
> >>> *********
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: visibility is
> >>> public,BD2,UNCLASSIFIED-NO_COMPARTMENT-UNRESTRICTED
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: inputDir is
> >>> /input
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: outputDir is
> >>> /output
> >>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
> >>> environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48
> >>> GMT
> >>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
> >>> environment:host.name=redhat-cloudbase1
> >>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client

You may be right about the tool.sh -libjars.  It doesn't copy the jar
files to the ${mapred.local.dir}/taskTracker/archive/ directory.
Neither does adding them to the HADOOP_CLASSPATH.

 

So to fix this problem I had to set up a DistributedCache and explicitly
add them to class path as follows:

 

DistributedCache.addFileToClassPath(new
Path("/usr/lib/accumulo-core-1.4.0.jar"), conf);

                DistributedCache.addFileToClassPath(new
Path("/usr/lib/kxml2-2.3.0.jar"), conf);

                DistributedCache.addFileToClassPath(new
Path("/usr/lib/xmlpull-1.1.3.1.jar"), conf);

                DistributedCache.addFileToClassPath(new
Path("/usr/lib/xstream-1.4.1.jar"), conf);

                DistributedCache.addFileToClassPath(new
Path("/usr/lib/zookeeper-3.4.3.jar"), conf);

                DistributedCache.addFileToClassPath(new
Path("/usr/lib/commons-collections-3.2.jar"), conf);

 

Although this works, it doesn't feel like I have something configured
correctly with my accumulo 1.4.0 upgrade.  Is there a better way to
configure the resources for the child jvm's?

 

From: John Vines [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, May 22, 2012 13:20
To: [EMAIL PROTECTED]
Subject: Re: AcculumoFileOutputFormat class cannot be found by child jvm

 

I'm wondering if our tool.sh plays nice with an explicit -libjars.

John

On Tue, May 22, 2012 at 2:07 PM, William Slacum <[EMAIL PROTECTED]>
wrote:

Make sure that the paths you pass to `-libjars` are correct. I have
noticed behavior with Hadoop where it will silently drop the end of a
libjars string if the paths do not exist.
On Tue, May 22, 2012 at 11:03 AM, Keith Turner <[EMAIL PROTECTED]> wrote:
> I think it should be in the job.xml file.  The way the class path for
> the java program that starts map reduce is configured is different
> from how the classpath for the remote task are configured.   So your
> program that starts the map reduce job has AccumuloFileOutputFormat on
> the class path because you use a static method on it and it runs.  So
> I am thinking its not in --libjars for some reason or maybe you are
> not using Hadoop Tool?
>
> On Tue, May 22, 2012 at 1:54 PM,  <[EMAIL PROTECTED]> wrote:
>> No, it's not in the job.xml file.  A reference to
AccumuloFileOutputFormat is but not the accumulo-core-1.4.0.jar.  The
Job*.jar file is referenced there.  I was under the impression that the
child jvm inherits the parent jvm attributes which makes the job*.jar
file.  Is that not correct?
>>
>> -----Original Message-----
>> From: Keith Turner [mailto:[EMAIL PROTECTED]]
>> Sent: Tuesday, May 22, 2012 12:07
>> To: [EMAIL PROTECTED]
>> Subject: Re: AcculumoFileOutputFormat class cannot be found by child
jvm
>>
>> If you look at Job.xml, do you see accumulo-core there?  There should
be a link to this file on the jobs page on the tasktracker.
>>
>> On Tue, May 22, 2012 at 10:40 AM,  <[EMAIL PROTECTED]> wrote:
>>> I upgrade to accumulo-1.4.0 and updated my map/reduce jobs and now
>>> they don't run.  The parent class path has the
accumulo-core-1.4.0.jar
>>> file included.  Do the accumulo jar files have to be manually put on
a
>>> distribute cache?  Any help is appreciated.
>>>
>>> [hadoop@redhat-cloudbase1 placemarks]$ ./runPlacemarkIngester.sh
Found
>>> 5 items drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:13
>>> /accumulo drwxrwxr-x   - hadoop hadoop          0 2012-05-21 15:06
>>> /data drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58
/input
>>> drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58 /output
>>> drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:34 /usr
Deleted
>>> hdfs://redhat-cloudbase1:9000/output
>>> Deleted hdfs://redhat-cloudbase1:9000/input
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: Invoking
>>> ToolRunner.run
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: zooKeeper is
>>> redhat-cloudbase1:2181
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: instanceName is
>>> NCCT-Cloudbase
is
metadataTableName
is
10:48
environment:java.class.path=/cloudbase/hadoop-0.20.2/bin/../conf:/usr/
va/jdk1.6.0_32/lib/tools.jar:/cloudbase/hadoop-0.20.2/bin/..:/cloudbas
hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar:/cloudbase/hadoop-0.20.2/b
/../lib/commons-cli-1.2.jar:/cloudbase/hadoop-0.20.2/bin/../lib/common
codec-1.3.jar:/cloudbase/hadoop-0.20.2/bin/../lib/commons-el-1.0.jar:/
oudbase/hadoop-0.20.2/bin/../lib/commons-httpclient-3.0.1.jar:/cloudba
/hadoop-0.20.2/bin/../lib/commons-logging-1.0.4.jar:/cloudbase/hadoop-0.
20.2/bin/../lib/commons-logging-api-1.0.4.jar:/cloudbase/hadoop-0.20.2
in/../lib/commons-net-1.4.1.jar:/cloudbase/hadoop-0.20.2/bin/../lib/co
-3.1.1.jar:/cloudbase/hadoop-0.20.2/bin/../lib/hadoop-0.20.2-ant.jar:/
oudbase/hadoop-0.20.2/bin/../lib/hadoop-0.20.2-core.jar:/cloudbase/had
p-0.20.2/bin/../lib/hadoop-0.20.2-examples.jar:/cloudbase/hadoop-0.20.
bin/../lib/hadoop-0.20.2-test.jar:/cloudbase/hadoop-0.20.2/bin/../lib/
doop-0.20.2-tools.jar:/cloudbase/hadoop-0.20.2/bin/../lib/hsqldb-1.8.0
0.jar:/cloudbase/hadoop-0.20.2/bin/../lib/jasper-compiler-5.5.12.jar:/
oudbase/hadoop-0.20.2/bin/../lib/jasper-runtime-5.5.12.jar:/cloudbase/
doop-0.20.2/bin/../lib/jets3t-0.6.1.jar:/cloudbase/hadoop-0.20.2/bin/.
lib/jetty-6.1.14.jar:/cloudbase/hadoop-0.20.2/bin/../lib/jetty-util-6.1.
14.jar:/cloudbase/hadoop-0.20.2/bin/../lib/junit-3.8.1.jar:/cloudbase/
doop-0.20.2/bin/../lib/kfs-0.2.2.jar:/cloudbase/hadoop-0.20.2/bin/../l
/log4j-1.2.15.jar:/cloudbase/hadoop-0.20.2/bin/../lib/mockito-all-1.8.0.
jar:/cloudbase/hadoop-0.20.2/bin/../lib/oro-2.0.8.jar:/cloudbase/hadoo
0.20.2/bin/../lib/servlet-api-2.5-6.1.14.jar:/cloudbase/hadoop-0.20.2/
n/../lib/slf4j-api-1.4.3.jar:/cloudbase/hadoop-0.20.2/bin/../lib/slf4j
og4j12-1.4.3.jar:/cloudbase/hadoop-0.20.2/bin/../lib/xmlenc-0.52.jar:/
oudbase/hadoop-0.20.2/bin/../lib/jsp-2.1/jsp-2.1.jar:/cloudbase/hadoop
.20.2/bin/../lib/jsp-2.1/jsp-api-2.1.jar:/cloudbase/accumulo-1.4.0/lib
ibthrift-0.6.1.jar:/cloudbase/accumulo-1.4

On Tuesday, May 22, 2012 2:19:34 PM, "John Vines" <[EMAIL PROTECTED]> wrote:
> I'm wondering if our tool.sh plays nice with an explicit -libjars.

It looks like tool.sh is attempting to parse explicit libjars, but it might not be working correctly.  The second-to-last line of tool.sh is

#echo exec "$HADOOP_HOME/bin/hadoop" jar "$TOOLJAR" $CLASSNAME -libjars \"$LIB_JARS\" $ARGS

If you could uncomment this line and send us what it displays, it might give us some insight into where tool.sh is going wrong.  The -libjars are supposed to include the user-specified libjars as well as accumulo-core, zookeeper, etc.

Billie
> John
>
>
> On Tue, May 22, 2012 at 2:07 PM, William Slacum < [EMAIL PROTECTED] >
> wrote:
>
>
> Make sure that the paths you pass to `-libjars` are correct. I have
> noticed behavior with Hadoop where it will silently drop the end of a
> libjars string if the paths do not exist.
>
>
>
> On Tue, May 22, 2012 at 11:03 AM, Keith Turner < [EMAIL PROTECTED] >
> wrote:
> > I think it should be in the job.xml file. The way the class path for
> > the java program that starts map reduce is configured is different
> > from how the classpath for the remote task are configured. So your
> > program that starts the map reduce job has AccumuloFileOutputFormat
> > on
> > the class path because you use a static method on it and it runs. So
> > I am thinking its not in --libjars for some reason or maybe you are
> > not using Hadoop Tool?
> >
> > On Tue, May 22, 2012 at 1:54 PM, < [EMAIL PROTECTED] > wrote:
> >> No, it's not in the job.xml file. A reference to
> >> AccumuloFileOutputFormat is but not the accumulo-core-1.4.0.jar.
> >> The Job*.jar file is referenced there. I was under the impression
> >> that the child jvm inherits the parent jvm attributes which makes
> >> the job*.jar file. Is that not correct?
> >>
> >> -----Original Message-----
> >> From: Keith Turner [mailto: [EMAIL PROTECTED] ]
> >> Sent: Tuesday, May 22, 2012 12:07
> >> To: [EMAIL PROTECTED]
> >> Subject: Re: AcculumoFileOutputFormat class cannot be found by
> >> child jvm
> >>
> >> If you look at Job.xml, do you see accumulo-core there? There
> >> should be a link to this file on the jobs page on the tasktracker.
> >>
> >> On Tue, May 22, 2012 at 10:40 AM, < [EMAIL PROTECTED] > wrote:
> >>> I upgrade to accumulo-1.4.0 and updated my map/reduce jobs and now
> >>> they don't run. The parent class path has the
> >>> accumulo-core-1.4.0.jar
> >>> file included. Do the accumulo jar files have to be manually put
> >>> on a
> >>> distribute cache? Any help is appreciated.
> >>>
> >>> [hadoop@redhat-cloudbase1 placemarks]$ ./runPlacemarkIngester.sh
> >>> Found
> >>> 5 items drwxrwxr-x - hadoop hadoop 0 2012-05-21 14:13
> >>> /accumulo drwxrwxr-x - hadoop hadoop 0 2012-05-21 15:06
> >>> /data drwxr-xr-x - hadoop hadoop 0 2012-05-22 08:58 /input
> >>> drwxr-xr-x - hadoop hadoop 0 2012-05-22 08:58 /output
> >>> drwxrwxr-x - hadoop hadoop 0 2012-05-21 14:34 /usr Deleted
> >>> hdfs://redhat-cloudbase1:9000/output
> >>> Deleted hdfs://redhat-cloudbase1:9000/input
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: Invoking
> >>> ToolRunner.run
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: zooKeeper is
> >>> redhat-cloudbase1:2181
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: instanceName
> >>> is
> >>> NCCT-Cloudbase
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: timeTableName
> >>> is
> >>> NCCTServicesTimes
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: geoTableName
> >>> is
> >>> NCCTServicesGeos
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester:
> >>> metadataTableName
> >>> is NCCTServicesMetadata
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: edgeTableName
> >>> is
> >>> NCCTEdgeTable
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: userName is
> >>> NCCT.Services.Client
> >>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: password is
> >>> *********

Here is the exec string.  As you can see the illusive jars are on the list, but the child jvm didn't get the memo on where they are.  The jars did not appear in the taskTracker or JobTracker directories.  The paths are correct to the jar files on the local file system.

exec /cloudbase/hadoop-0.20.2/bin/hadoop jar /mnt/hgfs/CSI.Cloudbase/Java/CloudbaseServices/out/artifacts/CloudbaseIngesters/CloudbaseIngesters.jar com.comcept.cloudbase.ingesters.placemarks.PlacemarkIngester -libjars "/cloudbase/accumulo-1.4.0/lib/libthrift-0.6.1.jar,/cloudbase/accumulo-1.4.0/lib/accumulo-core-1.4.0.jar,/cloudbase/zookeeper-3.4.3/zookeeper-3.4.3.jar,/cloudbase/accumulo-1.4.0/lib/cloudtrace-1.4.0.jar,/mnt/hgfs/CSI.Cloudbase/Java/CloudbaseServices/out/artifacts/CloudbaseIngesters/CloudbaseIngesters.jar,/usr/lib/ncct/kxml2-2.3.0.jar,/usr/lib/ncct/xmlpull-1.1.3.1.jar,/usr/lib/ncct/xstream-1.4.1.jar,/cloudbase/accumulo-1.4.0/lib/accumulo-core-1.4.0.jar,/cloudbase/accumulo-1.4.0/lib/commons-collections-3.2.jar,/cloudbase/accumulo-1.4.0/lib/commons-configuration-1.5.jar,/cloudbase/accumulo-1.4.0/lib/commons-io-1.4.jar,/cloudbase/accumulo-1.4.0/lib/commons-jci-core-1.0.jar,/cloudbase/accumulo-1.4.0/lib/commons-jci-fam-1.0.jar,/cloudbase/accumulo-1.4.0/lib/commons-lang-2.4.jar,/cloudbase/accumulo-1.4.0/lib/commons-logging-1.0.4.jar,/cloudbase/accumulo-1.4.0/lib/commons-logging-api-1.0.4.jar"

When I set up a DistributedCache on hdfs:// and added these jars to the job configuration the child jvm found them just fine.

-----Original Message-----
From: Billie J Rinaldi [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, May 22, 2012 16:50
To: [EMAIL PROTECTED]
Subject: Re: AcculumoFileOutputFormat class cannot be found by child jvm

On Tuesday, May 22, 2012 2:19:34 PM, "John Vines" <[EMAIL PROTECTED]> wrote:
> I'm wondering if our tool.sh plays nice with an explicit -libjars.

It looks like tool.sh is attempting to parse explicit libjars, but it might not be working correctly.  The second-to-last line of tool.sh is

#echo exec "$HADOOP_HOME/bin/hadoop" jar "$TOOLJAR" $CLASSNAME -libjars \"$LIB_JARS\" $ARGS

If you could uncomment this line and send us what it displays, it might give us some insight into where tool.sh is going wrong.  The -libjars are supposed to include the user-specified libjars as well as accumulo-core, zookeeper, etc.

Billie
> John
>
>
> On Tue, May 22, 2012 at 2:07 PM, William Slacum < [EMAIL PROTECTED] >
> wrote:
>
>
> Make sure that the paths you pass to `-libjars` are correct. I have
> noticed behavior with Hadoop where it will silently drop the end of a
> libjars string if the paths do not exist.
>
>
>
> On Tue, May 22, 2012 at 11:03 AM, Keith Turner < [EMAIL PROTECTED] >
> wrote:
> > I think it should be in the job.xml file. The way the class path for
> > the java program that starts map reduce is configured is different
> > from how the classpath for the remote task are configured. So your
> > program that starts the map reduce job has AccumuloFileOutputFormat
> > on the class path because you use a static method on it and it runs.
> > So I am thinking its not in --libjars for some reason or maybe you
> > are not using Hadoop Tool?
> >
> > On Tue, May 22, 2012 at 1:54 PM, < [EMAIL PROTECTED] > wrote:
> >> No, it's not in the job.xml file. A reference to
> >> AccumuloFileOutputFormat is but not the accumulo-core-1.4.0.jar.
> >> The Job*.jar file is referenced there. I was under the impression
> >> that the child jvm inherits the parent jvm attributes which makes
> >> the job*.jar file. Is that not correct?
> >>
> >> -----Original Message-----
> >> From: Keith Turner [mailto: [EMAIL PROTECTED] ]
> >> Sent: Tuesday, May 22, 2012 12:07
> >> To: [EMAIL PROTECTED]
> >> Subject: Re: AcculumoFileOutputFormat class cannot be found by
> >> child jvm
> >>
> >> If you look at Job.xml, do you see accumulo-core there? There
> >> should be a link to this file on the jobs page on the tasktracker.
> >>

On Tuesday, May 22, 2012 6:21:28 PM, "Bob Thorman" <[EMAIL PROTECTED]> wrote:
> Here is the exec string. As you can see the illusive jars are on the
> list, but the child jvm didn't get the memo on where they are. The
> jars did not appear in the taskTracker or JobTracker directories. The
> paths are correct to the jar files on the local file system.
>
> exec /cloudbase/hadoop-0.20.2/bin/hadoop jar
> /mnt/hgfs/CSI.Cloudbase/Java/CloudbaseServices/out/artifacts/CloudbaseIngesters/CloudbaseIngesters.jar
> com.comcept.cloudbase.ingesters.placemarks.PlacemarkIngester -libjars
> "/cloudbase/accumulo-1.4.0/lib/libthrift-0.6.1.jar,/cloudbase/accumulo-1.4.0/lib/accumulo-core-1.4.0.jar,/cloudbase/zookeeper-3.4.3/zookeeper-3.4.3.jar,/cloudbase/accumulo-1.4.0/lib/cloudtrace-1.4.0.jar,/mnt/hgfs/CSI.Cloudbase/Java/CloudbaseServices/out/artifacts/CloudbaseIngesters/CloudbaseIngesters.jar,/usr/lib/ncct/kxml2-2.3.0.jar,/usr/lib/ncct/xmlpull-1.1.3.1.jar,/usr/lib/ncct/xstream-1.4.1.jar,/cloudbase/accumulo-1.4.0/lib/accumulo-core-1.4.0.jar,/cloudbase/accumulo-1.4.0/lib/commons-collections-3.2.jar,/cloudbase/accumulo-1.4.0/lib/commons-configuration-1.5.jar,/cloudbase/accumulo-1.4.0/lib/commons-io-1.4.jar,/cloudbase/accumulo-1.4.0/lib/commons-jci-core-1.0.jar,/cloudbase/accumulo-1.4.0/lib/commons-jci-fam-1.0.jar,/cloudbase/accumulo-1.4.0/lib/commons-lang-2.4.jar,/cloudbase/accumulo-1.4.0/lib/commons-logging-1.0.4.jar,/cloudbase/accumulo-1.4.0/lib/commons-logging-api-1.0.4.jar"

The libjars string looks ok, assuming all those jars exist in those locations.  Can you identify what subset of these jars made it into the MapReduce config?  That might tell us where the error is occurring.

Billie

> When I set up a DistributedCache on hdfs:// and added these jars to
> the job configuration the child jvm found them just fine.
>
> -----Original Message-----
> From: Billie J Rinaldi [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, May 22, 2012 16:50
> To: [EMAIL PROTECTED]
> Subject: Re: AcculumoFileOutputFormat class cannot be found by child
> jvm
>
> On Tuesday, May 22, 2012 2:19:34 PM, "John Vines"
> <[EMAIL PROTECTED]> wrote:
> > I'm wondering if our tool.sh plays nice with an explicit -libjars.
>
> It looks like tool.sh is attempting to parse explicit libjars, but it
> might not be working correctly. The second-to-last line of tool.sh is
>
> #echo exec "$HADOOP_HOME/bin/hadoop" jar "$TOOLJAR" $CLASSNAME
> -libjars \"$LIB_JARS\" $ARGS
>
> If you could uncomment this line and send us what it displays, it
> might give us some insight into where tool.sh is going wrong. The
> -libjars are supposed to include the user-specified libjars as well as
> accumulo-core, zookeeper, etc.
>
> Billie
>
>
> > John
> >
> >
> > On Tue, May 22, 2012 at 2:07 PM, William Slacum < [EMAIL PROTECTED]
> > >
> > wrote:
> >
> >
> > Make sure that the paths you pass to `-libjars` are correct. I have
> > noticed behavior with Hadoop where it will silently drop the end of
> > a
> > libjars string if the paths do not exist.
> >
> >
> >
> > On Tue, May 22, 2012 at 11:03 AM, Keith Turner < [EMAIL PROTECTED] >
> > wrote:
> > > I think it should be in the job.xml file. The way the class path
> > > for
> > > the java program that starts map reduce is configured is different
> > > from how the classpath for the remote task are configured. So your
> > > program that starts the map reduce job has
> > > AccumuloFileOutputFormat
> > > on the class path because you use a static method on it and it
> > > runs.
> > > So I am thinking its not in --libjars for some reason or maybe you
> > > are not using Hadoop Tool?
> > >
> > > On Tue, May 22, 2012 at 1:54 PM, < [EMAIL PROTECTED] > wrote:
> > >> No, it's not in the job.xml file. A reference to
> > >> AccumuloFileOutputFormat is but not the accumulo-core-1.4.0.jar.
> > >> The Job*.jar file is referenced there. I was under the impression
> > >> that the child jvm inherits the parent jvm attributes which makes
> > >> the job*.jar file. Is that not correct?

Attached are the contents of the job.jar and job.xml and the taskTracker directory (job number changed because the GC removed the contents before I could copy/paste)

-----Original Message-----
From: Billie J Rinaldi [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, May 23, 2012 08:18
To: Thorman, Bob @ ISG - ComCept
Subject: Re: AcculumoFileOutputFormat class cannot be found by child jvm

On Tuesday, May 22, 2012 6:21:28 PM, "Bob Thorman" <[EMAIL PROTECTED]> wrote:
> Here is the exec string. As you can see the illusive jars are on the
> list, but the child jvm didn't get the memo on where they are. The
> jars did not appear in the taskTracker or JobTracker directories. The
> paths are correct to the jar files on the local file system.
>
> exec /cloudbase/hadoop-0.20.2/bin/hadoop jar
> /mnt/hgfs/CSI.Cloudbase/Java/CloudbaseServices/out/artifacts/Cloudbase
> Ingesters/CloudbaseIngesters.jar
> com.comcept.cloudbase.ingesters.placemarks.PlacemarkIngester -libjars
> "/cloudbase/accumulo-1.4.0/lib/libthrift-0.6.1.jar,/cloudbase/accumulo-1.4.0/lib/accumulo-core-1.4.0.jar,/cloudbase/zookeeper-3.4.3/zookeeper-3.4.3.jar,/cloudbase/accumulo-1.4.0/lib/cloudtrace-1.4.0.jar,/mnt/hgfs/CSI.Cloudbase/Java/CloudbaseServices/out/artifacts/CloudbaseIngesters/CloudbaseIngesters.jar,/usr/lib/ncct/kxml2-2.3.0.jar,/usr/lib/ncct/xmlpull-1.1.3.1.jar,/usr/lib/ncct/xstream-1.4.1.jar,/cloudbase/accumulo-1.4.0/lib/accumulo-core-1.4.0.jar,/cloudbase/accumulo-1.4.0/lib/commons-collections-3.2.jar,/cloudbase/accumulo-1.4.0/lib/commons-configuration-1.5.jar,/cloudbase/accumulo-1.4.0/lib/commons-io-1.4.jar,/cloudbase/accumulo-1.4.0/lib/commons-jci-core-1.0.jar,/cloudbase/accumulo-1.4.0/lib/commons-jci-fam-1.0.jar,/cloudbase/accumulo-1.4.0/lib/commons-lang-2.4.jar,/cloudbase/accumulo-1.4.0/lib/commons-logging-1.0.4.jar,/cloudbase/accumulo-1.4.0/lib/commons-logging-api-1.0.4.jar"

The libjars string looks ok, assuming all those jars exist in those locations.  Can you identify what subset of these jars made it into the MapReduce config?  That might tell us where the error is occurring.

Billie
> When I set up a DistributedCache on hdfs:// and added these jars to
> the job configuration the child jvm found them just fine.
>
> -----Original Message-----
> From: Billie J Rinaldi [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, May 22, 2012 16:50
> To: [EMAIL PROTECTED]
> Subject: Re: AcculumoFileOutputFormat class cannot be found by child
> jvm
>
> On Tuesday, May 22, 2012 2:19:34 PM, "John Vines"
> <[EMAIL PROTECTED]> wrote:
> > I'm wondering if our tool.sh plays nice with an explicit -libjars.
>
> It looks like tool.sh is attempting to parse explicit libjars, but it
> might not be working correctly. The second-to-last line of tool.sh is
>
> #echo exec "$HADOOP_HOME/bin/hadoop" jar "$TOOLJAR" $CLASSNAME
> -libjars \"$LIB_JARS\" $ARGS
>
> If you could uncomment this line and send us what it displays, it
> might give us some insight into where tool.sh is going wrong. The
> -libjars are supposed to include the user-specified libjars as well as
> accumulo-core, zookeeper, etc.
>
> Billie
>
>
> > John
> >
> >
> > On Tue, May 22, 2012 at 2:07 PM, William Slacum < [EMAIL PROTECTED]
> > >
> > wrote:
> >
> >
> > Make sure that the paths you pass to `-libjars` are correct. I have
> > noticed behavior with Hadoop where it will silently drop the end of
> > a libjars string if the paths do not exist.
> >
> >
> >
> > On Tue, May 22, 2012 at 11:03 AM, Keith Turner < [EMAIL PROTECTED] >
> > wrote:
> > > I think it should be in the job.xml file. The way the class path
> > > for the java program that starts map reduce is configured is
> > > different from how the classpath for the remote task are
> > > configured. So your program that starts the map reduce job has
> > > AccumuloFileOutputFormat on the class path because you use a
> > > static method on it and it runs.
> > > So I am thinking its not in --libjars for some reason or maybe you