Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> MR job runs on CDH3 but not on CDH4


Copy link to this message
-
Re: MR job runs on CDH3 but not on CDH4
It's hard for folks here to help you on CDH - please ask their own user lists.

Arun

On Jul 4, 2012, at 8:49 AM, Alan Miller wrote:

> Hi,
>  
> I’m trying to move from CDH3U3 to CDH4.
> My existing MR program works fine on CDH3U3  but I cant get it to run on CDH4.
>  
> Basically my Driver class
> 1.       queries a PG DB and writes some HashMaps to files in the Distributed Cache,
> 2.       then writes some Avro files (avro 1.7.0) to HDFS,
> 3.       and then triggers a MRv1 job to process the Avro files.
>  
> The DC & Avro files get written so HDFS is working, but my job is not getting started.
> I get an error:
> Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy()      
> …
>                 at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:478)
>  
> Seems the job doesn’t even get accepted .
> At  MyDriver:397 (see below) I call job.submit, but that’s it.
>  
> …..
> 12/07/04 08:16:54 INFO MyDriver: Processing data [my-data]
> 12/07/04 08:16:54 INFO MyDriver: Write Avro: [Tue Jul 03 00:00:00 PDT 2012 > etime <= Tue Jul 03 23:59:59 PDT 2012]
> 12/07/04 08:19:47 INFO MyDriver: Initialized file /data/in/my-data_2012-07-03.avro
> 12/07/04 08:19:49 INFO MyDriver: Read 158285 lines, Wrote 158131 records to 1 file(s)
> 12/07/04 08:19:49 INFO MyDriver: Wed Jul 04 08:19:49 PDT 2012 Finished avro data /data/in/my-data_2012-07-03.avro
> 12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/83206b5c-8a1c-46f3-bfb2-d8c3e949a530#q_map to distributed cache.
> 12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/b2ebfeb9-bdb0-489e-8186-8e18f4416224#u_map to distributed cache.
> 12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/437cfd91-aa07-4c3a-b4c9-cd4ae076f7ad#r_map to distributed cache.
> 12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/9554fe48-2171-423c-ba54-6249ffc882d4#m_map to distributed cache.
> 12/07/04 08:19:49 INFO MyDriver: Added /data/in/y-data_2012-07-03.avro to input files list.
> Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/security/UserGroupInformation;Lorg/apache/hadoop/conf/Configuration;Ljavax/net/SocketFactory;)Lorg/apache/hadoop/ipc/VersionedProtocol;
>                 at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:478)
>                 at org.apache.hadoop.mapred.JobClient.init(JobClient.java:472)
>                 at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:455)
>                 at org.apache.hadoop.mapreduce.Job$1.run(Job.java:478)
>                 at java.security.AccessController.doPrivileged(Native Method)
>                 at javax.security.auth.Subject.doAs(Subject.java:396)
>                 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>                 at org.apache.hadoop.mapreduce.Job.connect(Job.java:476)
>                 at org.apache.hadoop.mapreduce.Job.submit(Job.java:464)
>                 at com.mycompany.MyDriver.run(MyDriver.java:397)
>                 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>                 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>                 at com.mycompany.MyDriver.runHadoopJob(MyDriver.java:308)
>                 at com.mycompany.MyDriver.main(MyDriver.java:1532)
>                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>                 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>                 at java.lang.reflect.Method.invoke(Method.java:597)
>                 at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>  
>  
> Alan

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB