Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: InvalidProtocolBufferException while submitting crunch job to cluster


Copy link to this message
-
Re: InvalidProtocolBufferException while submitting crunch job to cluster
: java.net.UnknownHostException: bdatadev
edit your /etc/hosts file
Regards,
Som Shekhar Sharma
+91-8197243810
On Sat, Aug 31, 2013 at 2:05 AM, Narlin M <[EMAIL PROTECTED]> wrote:
> Looks like I was pointing to incorrect ports. After correcting the port
> numbers,
>
> conf.set("fs.defaultFS", "hdfs://<server_address>:8020");
> conf.set("mapred.job.tracker", "<server_address>:8021");
>
> I am now getting the following exception:
>
> 2880 [Thread-15] INFO
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob  -
> java.lang.IllegalArgumentException: java.net.UnknownHostException: bdatadev
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:414)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:164)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:389)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:356)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:124)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2218)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2252)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2234)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:300)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
> at
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:103)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:902)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:896)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:531)
> at
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.submit(CrunchControlledJob.java:305)
> at
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.startReadyJobs(CrunchJobControl.java:180)
> at
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.pollJobStatusAndStartNewOnes(CrunchJobControl.java:209)
> at
> org.apache.crunch.impl.mr.exec.MRExecutor.monitorLoop(MRExecutor.java:100)
> at org.apache.crunch.impl.mr.exec.MRExecutor.access$000(MRExecutor.java:51)
> at org.apache.crunch.impl.mr.exec.MRExecutor$1.run(MRExecutor.java:75)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.net.UnknownHostException: bdatadev
> ... 27 more
>
> However nowhere in my code a host named "bdatadev" is mentioned, and I
> cannot ping this host.
>
> Thanks for the help.
>
>
> On Fri, Aug 30, 2013 at 3:04 PM, Narlin M <[EMAIL PROTECTED]> wrote:
>>
>> I am getting following exception while trying to submit a crunch pipeline
>> job to a remote hadoop cluster:
>>
>> Exception in thread "main" java.lang.RuntimeException: Cannot create job
>> output directory /tmp/crunch-324987940
>> at
>> org.apache.crunch.impl.mr.MRPipeline.createTempDirectory(MRPipeline.java:344)
>> at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:125)
>> at test.CrunchTest.setup(CrunchTest.java:98)
>> at test.CrunchTest.main(CrunchTest.java:367)
>> Caused by: java.io.IOException: Failed on local exception:
>> com.google.protobuf.InvalidProtocolBufferException: Protocol message
>> end-group tag did not match expected tag.; Host Details : local host is:
>> "NARLIN/127.0.0.1"; destination host is: "<server_address>":50070;
>> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:759)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1164)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
>> at com.sun.proxy.$Proxy11.mkdirs(Unknown Source)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB