Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: InvalidProtocolBufferException while submitting crunch job to cluster


Copy link to this message
-
Re: InvalidProtocolBufferException while submitting crunch job to cluster
: java.net.UnknownHostException: bdatadev
edit your /etc/hosts file
Regards,
Som Shekhar Sharma
+91-8197243810
On Sat, Aug 31, 2013 at 2:05 AM, Narlin M <[EMAIL PROTECTED]> wrote:
> Looks like I was pointing to incorrect ports. After correcting the port
> numbers,
>
> conf.set("fs.defaultFS", "hdfs://<server_address>:8020");
> conf.set("mapred.job.tracker", "<server_address>:8021");
>
> I am now getting the following exception:
>
> 2880 [Thread-15] INFO
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob  -
> java.lang.IllegalArgumentException: java.net.UnknownHostException: bdatadev
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:414)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:164)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:389)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:356)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:124)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2218)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2252)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2234)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:300)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
> at
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:103)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:902)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:896)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:531)
> at
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.submit(CrunchControlledJob.java:305)
> at
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.startReadyJobs(CrunchJobControl.java:180)
> at
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.pollJobStatusAndStartNewOnes(CrunchJobControl.java:209)
> at
> org.apache.crunch.impl.mr.exec.MRExecutor.monitorLoop(MRExecutor.java:100)
> at org.apache.crunch.impl.mr.exec.MRExecutor.access$000(MRExecutor.java:51)
> at org.apache.crunch.impl.mr.exec.MRExecutor$1.run(MRExecutor.java:75)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.net.UnknownHostException: bdatadev
> ... 27 more
>
> However nowhere in my code a host named "bdatadev" is mentioned, and I
> cannot ping this host.
>
> Thanks for the help.
>
>
> On Fri, Aug 30, 2013 at 3:04 PM, Narlin M <[EMAIL PROTECTED]> wrote:
>>
>> I am getting following exception while trying to submit a crunch pipeline
>> job to a remote hadoop cluster:
>>
>> Exception in thread "main" java.lang.RuntimeException: Cannot create job
>> output directory /tmp/crunch-324987940
>> at
>> org.apache.crunch.impl.mr.MRPipeline.createTempDirectory(MRPipeline.java:344)
>> at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:125)
>> at test.CrunchTest.setup(CrunchTest.java:98)
>> at test.CrunchTest.main(CrunchTest.java:367)
>> Caused by: java.io.IOException: Failed on local exception:
>> com.google.protobuf.InvalidProtocolBufferException: Protocol message
>> end-group tag did not match expected tag.; Host Details : local host is:
>> "NARLIN/127.0.0.1"; destination host is: "<server_address>":50070;
>> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:759)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1164)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
>> at com.sun.proxy.$Proxy11.mkdirs(Unknown Source)