Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # user >> AcculumoFileOutputFormat class cannot be found by child jvm


Copy link to this message
-
RE: AcculumoFileOutputFormat class cannot be found by child jvm
You may be right about the tool.sh -libjars.  It doesn't copy the jar
files to the ${mapred.local.dir}/taskTracker/archive/ directory.
Neither does adding them to the HADOOP_CLASSPATH.

 

So to fix this problem I had to set up a DistributedCache and explicitly
add them to class path as follows:

 

DistributedCache.addFileToClassPath(new
Path("/usr/lib/accumulo-core-1.4.0.jar"), conf);

                DistributedCache.addFileToClassPath(new
Path("/usr/lib/kxml2-2.3.0.jar"), conf);

                DistributedCache.addFileToClassPath(new
Path("/usr/lib/xmlpull-1.1.3.1.jar"), conf);

                DistributedCache.addFileToClassPath(new
Path("/usr/lib/xstream-1.4.1.jar"), conf);

                DistributedCache.addFileToClassPath(new
Path("/usr/lib/zookeeper-3.4.3.jar"), conf);

                DistributedCache.addFileToClassPath(new
Path("/usr/lib/commons-collections-3.2.jar"), conf);

 

Although this works, it doesn't feel like I have something configured
correctly with my accumulo 1.4.0 upgrade.  Is there a better way to
configure the resources for the child jvm's?

 

From: John Vines [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, May 22, 2012 13:20
To: [EMAIL PROTECTED]
Subject: Re: AcculumoFileOutputFormat class cannot be found by child jvm

 

I'm wondering if our tool.sh plays nice with an explicit -libjars.

John

On Tue, May 22, 2012 at 2:07 PM, William Slacum <[EMAIL PROTECTED]>
wrote:

Make sure that the paths you pass to `-libjars` are correct. I have
noticed behavior with Hadoop where it will silently drop the end of a
libjars string if the paths do not exist.
On Tue, May 22, 2012 at 11:03 AM, Keith Turner <[EMAIL PROTECTED]> wrote:
> I think it should be in the job.xml file.  The way the class path for
> the java program that starts map reduce is configured is different
> from how the classpath for the remote task are configured.   So your
> program that starts the map reduce job has AccumuloFileOutputFormat on
> the class path because you use a static method on it and it runs.  So
> I am thinking its not in --libjars for some reason or maybe you are
> not using Hadoop Tool?
>
> On Tue, May 22, 2012 at 1:54 PM,  <[EMAIL PROTECTED]> wrote:
>> No, it's not in the job.xml file.  A reference to
AccumuloFileOutputFormat is but not the accumulo-core-1.4.0.jar.  The
Job*.jar file is referenced there.  I was under the impression that the
child jvm inherits the parent jvm attributes which makes the job*.jar
file.  Is that not correct?
>>
>> -----Original Message-----
>> From: Keith Turner [mailto:[EMAIL PROTECTED]]
>> Sent: Tuesday, May 22, 2012 12:07
>> To: [EMAIL PROTECTED]
>> Subject: Re: AcculumoFileOutputFormat class cannot be found by child
jvm
>>
>> If you look at Job.xml, do you see accumulo-core there?  There should
be a link to this file on the jobs page on the tasktracker.
>>
>> On Tue, May 22, 2012 at 10:40 AM,  <[EMAIL PROTECTED]> wrote:
>>> I upgrade to accumulo-1.4.0 and updated my map/reduce jobs and now
>>> they don't run.  The parent class path has the
accumulo-core-1.4.0.jar
>>> file included.  Do the accumulo jar files have to be manually put on
a
>>> distribute cache?  Any help is appreciated.
>>>
>>> [hadoop@redhat-cloudbase1 placemarks]$ ./runPlacemarkIngester.sh
Found
>>> 5 items drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:13
>>> /accumulo drwxrwxr-x   - hadoop hadoop          0 2012-05-21 15:06
>>> /data drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58
/input
>>> drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58 /output
>>> drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:34 /usr
Deleted
>>> hdfs://redhat-cloudbase1:9000/output
>>> Deleted hdfs://redhat-cloudbase1:9000/input
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: Invoking
>>> ToolRunner.run
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: zooKeeper is
>>> redhat-cloudbase1:2181
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: instanceName is
>>> NCCT-Cloudbase
is
metadataTableName
is
10:48
environment:java.class.path=/cloudbase/hadoop-0.20.2/bin/../conf:/usr/
va/jdk1.6.0_32/lib/tools.jar:/cloudbase/hadoop-0.20.2/bin/..:/cloudbas
hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar:/cloudbase/hadoop-0.20.2/b
/../lib/commons-cli-1.2.jar:/cloudbase/hadoop-0.20.2/bin/../lib/common
codec-1.3.jar:/cloudbase/hadoop-0.20.2/bin/../lib/commons-el-1.0.jar:/
oudbase/hadoop-0.20.2/bin/../lib/commons-httpclient-3.0.1.jar:/cloudba
/hadoop-0.20.2/bin/../lib/commons-logging-1.0.4.jar:/cloudbase/hadoop-0.
20.2/bin/../lib/commons-logging-api-1.0.4.jar:/cloudbase/hadoop-0.20.2
in/../lib/commons-net-1.4.1.jar:/cloudbase/hadoop-0.20.2/bin/../lib/co
-3.1.1.jar:/cloudbase/hadoop-0.20.2/bin/../lib/hadoop-0.20.2-ant.jar:/
oudbase/hadoop-0.20.2/bin/../lib/hadoop-0.20.2-core.jar:/cloudbase/had
p-0.20.2/bin/../lib/hadoop-0.20.2-examples.jar:/cloudbase/hadoop-0.20.
bin/../lib/hadoop-0.20.2-test.jar:/cloudbase/hadoop-0.20.2/bin/../lib/
doop-0.20.2-tools.jar:/cloudbase/hadoop-0.20.2/bin/../lib/hsqldb-1.8.0
0.jar:/cloudbase/hadoop-0.20.2/bin/../lib/jasper-compiler-5.5.12.jar:/
oudbase/hadoop-0.20.2/bin/../lib/jasper-runtime-5.5.12.jar:/cloudbase/
doop-0.20.2/bin/../lib/jets3t-0.6.1.jar:/cloudbase/hadoop-0.20.2/bin/.
lib/jetty-6.1.14.jar:/cloudbase/hadoop-0.20.2/bin/../lib/jetty-util-6.1.
14.jar:/cloudbase/hadoop-0.20.2/bin/../lib/junit-3.8.1.jar:/cloudbase/
doop-0.20.2/bin/../lib/kfs-0.2.2.jar:/cloudbase/hadoop-0.20.2/bin/../l
/log4j-1.2.15.jar:/cloudbase/hadoop-0.20.2/bin/../lib/mockito-all-1.8.0.
jar:/cloudbase/hadoop-0.20.2/bin/../lib/oro-2.0.8.jar:/cloudbase/hadoo
0.20.2/bin/../lib/servlet-api-2.5-6.1.14.jar:/cloudbase/hadoop-0.20.2/
n/../lib/slf4j-api-1.4.3.jar:/cloudbase/hadoop-0.20.2/bin/../lib/slf4j
og4j12-1.4.3.jar:/cloudbase/hadoop-0.20.2/bin/../lib/xmlenc-0.52.jar:/
oudbase/hadoop-0.20.2/bin/../lib/jsp-2.1/jsp-2.1.jar:/cloudbase/hadoop
.20.2/bin/../lib/jsp-2.1/jsp-api-2.1.jar:/cloudbase/accumulo-1.4.0/lib
ibthrift-0.6.1.jar:/cloudbase/accumulo-1.4
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB