Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> SpecificData.deepCopy exception with CDH4


+
Jeremy Lewi 2012-07-15, 01:59
+
Alan Miller 2012-07-15, 09:00
Copy link to this message
-
Re: SpecificData.deepCopy exception with CDH4
Thanks Alan.

I'm still getting the same error as before. Here's how I'm running the job
*

   -

*
hadoop jar ./target/contrail-1.0-SNAPSHOT-job.jar
contrail.avro.QuickMergeAvro -D mapreduce.task.classpath.first=true
-libjars=/users/jlewi/svn_avro_1.6.1/lang/java/avro/target/avro-1.6.1.jar,/users/jlewi/svn_avro_1.6.1/lang/java/mapred/target/avro-mapred-1.6.1.jar
--inputpath=/users/jlewi/staph/assembly/BuildGraph
--outputpath=/users/jlewi/staph/assembly/QuickMerge --K=45

I verified via the job tracker that the property
"mapreduce.task.classpath.first"  is getting picked up.

It looks like the problem I'm dealing with is related to
https://issues.apache.org/jira/browse/AVRO-1103.

Any ideas?

Thanks
J
On Sun, Jul 15, 2012 at 2:00 AM, Alan Miller <[EMAIL PROTECTED]>wrote:

> Hi Just a quick idea.
> Also check ALL directories returned by
>   hadoop classpath
> for any Avro related classes.
>
> I was struggling trying to use
> avro-1.7.0 with CDH4 but made it work
> by using the -libjars option and making sure my classes are used BEFORE
> the standard classes. There's a config
> property (dont remember) to set for
> that. Note the above setting is for the
> task's classpath, to control the
> classpath of your driver class set
> HADOOOP_CLASSPATH=... and
> HADOOOP_USER_CLASSPATH_FIRST=true
>
> Alan
>
> Sent from my iPhone
>
> On Jul 15, 2012, at 3:59, "Jeremy Lewi" <[EMAIL PROTECTED]> wrote:
>
> > hi avro-users,
> >
> > I'm getting the following exception when using avro 1.6.1 with CDH4.
> > java.lang.NoSuchMethodError:
> org.apache.avro.specific.SpecificData.deepCopy(Lorg/apache/avro/Schema;Ljava/lang/Object;)Ljava/lang/Object;
> >
> > The offending code is
> > GraphNodeData copy = (GraphNodeData)
> SpecificData.get().deepCopy(data.getSchema(), data);
> >
> > where GraphNodeData is a class generated from my AVRO record.
> >
> > The code runs just fine on CDH3. I tried rebuilding AVRO from source and
> installing it my local repo because of a previous post that said Avro 1.6.1
> in maven had been built against CDH3. I also deleted all the avro jar files
> I found in
> > /usr/lib/hadoop
> >
> > Any ideas? Thanks?
> > Jeremy
> >
> >
> >
>
+
Jeremy Lewi 2012-07-16, 07:17
+
Jeremy Lewi 2012-07-16, 15:19
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB