Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> InvalidProtocolBufferException while submitting crunch job to cluster

Narlin M 2013-08-30, 20:04
Copy link to this message
Re: InvalidProtocolBufferException while submitting crunch job to cluster
Your cluster is using HDFS HA, and therefore requires a little more
configs than just fs.defaultFS/etc..

You need to use the right set of cluster client configs. If you don't
have them at /etc/hadoop/conf and /etc/hbase/conf on your cluster edge
node to pull from, try asking your cluster administrator for a
configuration set, and place their parent directories on your
application's classpath.

The first error deals with perhaps you also including a guava
dependency in your project, which is different than the one
transitively pulled in by hadoop-client via crunch. You should be able
to use guava libs without needing an explicit dependency, and it would
be the right needed version.

The second error deals with your MR submission failing, cause the JT
is using a staging directory over a HDFS HA, which uses a "logical"
name of "bdatadev". A logical HA name needs other configs (typically
in the hdfs-site.xml) that tell it which are the actual physical NNs
under it - configs that you're missing here.

On Sat, Aug 31, 2013 at 1:34 AM, Narlin M <[EMAIL PROTECTED]> wrote:
> I am getting following exception while trying to submit a crunch pipeline
> job to a remote hadoop cluster:
> Exception in thread "main" java.lang.RuntimeException: Cannot create job
> output directory /tmp/crunch-324987940
> at
> org.apache.crunch.impl.mr.MRPipeline.createTempDirectory(MRPipeline.java:344)
> at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:125)
> at test.CrunchTest.setup(CrunchTest.java:98)
> at test.CrunchTest.main(CrunchTest.java:367)
> Caused by: java.io.IOException: Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Protocol message
> end-group tag did not match expected tag.; Host Details : local host is:
> "NARLIN/"; destination host is: "<server_address>":50070;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:759)
> at org.apache.hadoop.ipc.Client.call(Client.java:1164)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> at com.sun.proxy.$Proxy11.mkdirs(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
> at com.sun.proxy.$Proxy11.mkdirs(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:425)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:1943)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:523)
> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1799)
> at
> org.apache.crunch.impl.mr.MRPipeline.createTempDirectory(MRPipeline.java:342)
> ... 3 more
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
> message end-group tag did not match expected tag.
> at
> com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:73)
> at
> com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
> at
> com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:213)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:746)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238)
> at
> com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:282)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:760)
> at
> com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:288)
> at

Harsh J
Narlin M 2013-09-03, 13:36
Narlin M 2013-08-31, 13:38