Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - MR job runs on CDH3 but not on CDH4


+
Alan Miller 2012-07-04, 15:49
Copy link to this message
-
Re: MR job runs on CDH3 but not on CDH4
Arun C Murthy 2012-07-04, 17:21
It's hard for folks here to help you on CDH - please ask their own user lists.

Arun

On Jul 4, 2012, at 8:49 AM, Alan Miller wrote:

> Hi,
>  
> I’m trying to move from CDH3U3 to CDH4.
> My existing MR program works fine on CDH3U3  but I cant get it to run on CDH4.
>  
> Basically my Driver class
> 1.       queries a PG DB and writes some HashMaps to files in the Distributed Cache,
> 2.       then writes some Avro files (avro 1.7.0) to HDFS,
> 3.       and then triggers a MRv1 job to process the Avro files.
>  
> The DC & Avro files get written so HDFS is working, but my job is not getting started.
> I get an error:
> Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy()      
> …
>                 at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:478)
>  
> Seems the job doesn’t even get accepted .
> At  MyDriver:397 (see below) I call job.submit, but that’s it.
>  
> …..
> 12/07/04 08:16:54 INFO MyDriver: Processing data [my-data]
> 12/07/04 08:16:54 INFO MyDriver: Write Avro: [Tue Jul 03 00:00:00 PDT 2012 > etime <= Tue Jul 03 23:59:59 PDT 2012]
> 12/07/04 08:19:47 INFO MyDriver: Initialized file /data/in/my-data_2012-07-03.avro
> 12/07/04 08:19:49 INFO MyDriver: Read 158285 lines, Wrote 158131 records to 1 file(s)
> 12/07/04 08:19:49 INFO MyDriver: Wed Jul 04 08:19:49 PDT 2012 Finished avro data /data/in/my-data_2012-07-03.avro
> 12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/83206b5c-8a1c-46f3-bfb2-d8c3e949a530#q_map to distributed cache.
> 12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/b2ebfeb9-bdb0-489e-8186-8e18f4416224#u_map to distributed cache.
> 12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/437cfd91-aa07-4c3a-b4c9-cd4ae076f7ad#r_map to distributed cache.
> 12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/9554fe48-2171-423c-ba54-6249ffc882d4#m_map to distributed cache.
> 12/07/04 08:19:49 INFO MyDriver: Added /data/in/y-data_2012-07-03.avro to input files list.
> Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/security/UserGroupInformation;Lorg/apache/hadoop/conf/Configuration;Ljavax/net/SocketFactory;)Lorg/apache/hadoop/ipc/VersionedProtocol;
>                 at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:478)
>                 at org.apache.hadoop.mapred.JobClient.init(JobClient.java:472)
>                 at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:455)
>                 at org.apache.hadoop.mapreduce.Job$1.run(Job.java:478)
>                 at java.security.AccessController.doPrivileged(Native Method)
>                 at javax.security.auth.Subject.doAs(Subject.java:396)
>                 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>                 at org.apache.hadoop.mapreduce.Job.connect(Job.java:476)
>                 at org.apache.hadoop.mapreduce.Job.submit(Job.java:464)
>                 at com.mycompany.MyDriver.run(MyDriver.java:397)
>                 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>                 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>                 at com.mycompany.MyDriver.runHadoopJob(MyDriver.java:308)
>                 at com.mycompany.MyDriver.main(MyDriver.java:1532)
>                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>                 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>                 at java.lang.reflect.Method.invoke(Method.java:597)
>                 at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>  
>  
> Alan

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
+
Alan Miller 2012-07-04, 19:32