Avro 17 is compiled against Hadoop 1 i.e. CDH3.

I have it running aginst CDH4 but I had to patch and recompile Avro, then replace the Avro 153 in the Hadoop lib directory with 17. I have attached my patch against JIRA AVRO-1103 for consideration.

-----Original Message-----

From: Alan Miller
Sent: 5 Jul 2012 11:03:06 GMT
To: [EMAIL PROTECTED]
Subject: Re: avro 1.7.0 with CDH4
I noticed that CDH4 has avro-1.5.4.jar in hadoop/lib so I modified my job so
that:

 1.  avro-1.7.0.jar and avro-tools-1.7.0.jar is contained within my jar file
in ./lib/
    (note: I submit my job via hadoop jar myjar.jar myDriver <myopts ....>)
 2.  I have also set: mapreduce.task.classpath.first=true (in
mapred-site.xml)
      to ensure the 1.7.0 jars from the lib dir in my.jar are used first
(before avro-1.5.4).

But I still get the same error.

Any ideas?

Alan

--
View this message in context: http://apache-avro.679487.n3.nabble.com/avro-1-7-0-with-CDH4-tp4025149p4025150.html
Sent from the Avro - Users mailing list archive at Nabble.com.

What is the correct solution for making avro-1.7 work with both Hadoop 1
(CDH3) and Hadoop 2 (CDH4)? Should we add build profiles for each so we can
release two separate avro-mapred jars?

On Thu, Jul 5, 2012 at 5:12 AM, Jacob Metcalf <[EMAIL PROTECTED]>wrote:

> Avro 17 is compiled against Hadoop 1 i.e. CDH3.
>
> I have it running aginst CDH4 but I had to patch and recompile Avro, then
> replace the Avro 153 in the Hadoop lib directory with 17. I have attached
> my patch against JIRA AVRO-1103 for consideration.
>
> -----Original Message-----
>
> From: Alan Miller
> Sent: 5 Jul 2012 11:03:06 GMT
> To: [EMAIL PROTECTED]
> Subject: Re: avro 1.7.0 with CDH4
>
>
> I noticed that CDH4 has avro-1.5.4.jar in hadoop/lib so I modified my job
> so
> that:
>
>  1.  avro-1.7.0.jar and avro-tools-1.7.0.jar is contained within my jar
> file
> in ./lib/
>     (note: I submit my job via hadoop jar myjar.jar myDriver <myopts ....>)
>  2.  I have also set: mapreduce.task.classpath.first=true (in
> mapred-site.xml)
>       to ensure the 1.7.0 jars from the lib dir in my.jar are used first
> (before avro-1.5.4).
>
> But I still get the same error.
>
> Any ideas?
>
> Alan
>
> --
> View this message in context:
> http://apache-avro.679487.n3.nabble.com/avro-1-7-0-with-CDH4-tp4025149p4025150.html
> Sent from the Avro - Users mailing list archive at Nabble.com.
>

Thanks for the 1.7.0 release.
I was waiting for AVRO-593 so I can use the new mapreduce API with Avro.

Previously, I was using avro-1.6.3 and odiago-avro-1.0.7.jar
and everything was working. My driver class writes Avro files on HDFS
and triggers an AvroJob to process them.

I recompiled my code with avro-1.7.0 (and removed odiago-avro) but now I get
the error below. I know this list is about Avro and not odiago-avro
integration but
is AvroKeyInputFormat from avro-1.7.0-tools compatible with odiago's?
2012-07-05 02:21:10,068 FATAL org.apache.hadoop.mapred.Child: Error running
child : java.lang.IncompatibleClassChangeError: Found interface
org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
at
org.apache.avro.mapreduce.AvroKeyInputFormat.createRecordReader(AvroKeyInputFormat.java:46)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:617)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.Child.main(Child.java:264)

--
View this message in context: http://apache-avro.679487.n3.nabble.com/avro-1-7-0-with-CDH4-tp4025149.html
Sent from the Avro - Users mailing list archive at Nabble.com.

I noticed that CDH4 has avro-1.5.4.jar in hadoop/lib so I modified my job so
that:

 1.  avro-1.7.0.jar and avro-tools-1.7.0.jar is contained within my jar file
in ./lib/
    (note: I submit my job via hadoop jar myjar.jar myDriver <myopts ....>)
 2.  I have also set: mapreduce.task.classpath.first=true (in
mapred-site.xml)
      to ensure the 1.7.0 jars from the lib dir in my.jar are used first
(before avro-1.5.4).

But I still get the same error.

Any ideas?

Alan

--
View this message in context: http://apache-avro.679487.n3.nabble.com/avro-1-7-0-with-CDH4-tp4025149p4025150.html
Sent from the Avro - Users mailing list archive at Nabble.com.

That would be ideal. With MRUnit for example both builds are in the repo and you use <classifier>hadoop2</classifier> to get the right one.

-----Original Message-----

From: Garrett Wu
Sent: 16 Jul 2012 16:56:54 GMT
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: Re: avro 1.7.0 with CDH4

What is the correct solution for making avro-1.7 work with both Hadoop 1
(CDH3) and Hadoop 2 (CDH4)? Should we add build profiles for each so we can
release two separate avro-mapred jars?

On Thu, Jul 5, 2012 at 5:12 AM, Jacob Metcalf <[EMAIL PROTECTED]>wrote:

> Avro 17 is compiled against Hadoop 1 i.e. CDH3.
>
> I have it running aginst CDH4 but I had to patch and recompile Avro, then
> replace the Avro 153 in the Hadoop lib directory with 17. I have attached
> my patch against JIRA AVRO-1103 for consideration.
>
> -----Original Message-----
>
> From: Alan Miller
> Sent: 5 Jul 2012 11:03:06 GMT
> To: [EMAIL PROTECTED]
> Subject: Re: avro 1.7.0 with CDH4
>
>
> I noticed that CDH4 has avro-1.5.4.jar in hadoop/lib so I modified my job
> so
> that:
>
>  1.  avro-1.7.0.jar and avro-tools-1.7.0.jar is contained within my jar
> file
> in ./lib/
>     (note: I submit my job via hadoop jar myjar.jar myDriver <myopts ....>)
>  2.  I have also set: mapreduce.task.classpath.first=true (in
> mapred-site.xml)
>       to ensure the 1.7.0 jars from the lib dir in my.jar are used first
> (before avro-1.5.4).
>
> But I still get the same error.
>
> Any ideas?
>
> Alan
>
> --
> View this message in context:
> http://apache-avro.679487.n3.nabble.com/avro-1-7-0-with-CDH4-tp4025149p4025150.html
> Sent from the Avro - Users mailing list archive at Nabble.com.
>