Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Re: Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" on writing to HDFS sink


Copy link to this message
-
Re: Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" on writing to HDFS sink
Glad to hear that was the issue.

dave
On Tue, Oct 29, 2013 at 12:53 AM, wesley dias <[EMAIL PROTECTED]>wrote:

> Thanks Roshan. But issue is now resolved. I compiled flume 1.4 from source
> using protobuf 2.5 and this solved the below mentioned issue. My guess is
> that the distribution available for flume 1.4 from the release site must
> have been compiled using protobuf 2.4
>
> Regards,
>
> Wesley
>
> ------------------------------
> Date: Mon, 28 Oct 2013 13:41:52 -0700
>
> Subject: Re: Exception in thread
> "SinkRunner-PollingRunner-DefaultSinkProcessor" on writing to HDFS sink
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
>
> HDP already includes hadoop 2.2.0 + Flume 1.4. You'd be better of just
> using instead of building and testing that combo out on your own.
>
> http://hortonworks.com/products/hdp-2/?b=2#overview
>
>
> On Mon, Oct 28, 2013 at 1:19 PM, wesley dias <[EMAIL PROTECTED]>wrote:
>
> Hi David,
>
> Thanks for the quick reply.Does this mean that I would need to compile the
> flume from source using protobuf 2.5 instead of directly using the flume
> distribution available for download from the site? Is flume 1.4 based of
> protobuf 2.4?
> I am not sure of the same hence I am asking these questions.
>
> Regards,
>
> Wesley
> ------------------------------
> Date: Mon, 28 Oct 2013 16:11:16 -0400
> Subject: Re: Exception in thread
> "SinkRunner-PollingRunner-DefaultSinkProcessor" on writing to HDFS sink
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
>
>
> Looks like you have a protocol buffer version problem. Do you have a
> dependency on 2.5.0 version of protocol buffers in your pom?
>
> see
>
> http://code.google.com/p/protobuf/issues/detail?id=493
>
>
>
> On Mon, Oct 28, 2013 at 4:02 PM, wesley dias <[EMAIL PROTECTED]>wrote:
>
> Hi,
>
> I am using the latest flume build 1.4 along with hadoop 2.2.0 to collect
> twitter stream and write to HDFS sink. However, on writing to HDFS I get
> the following error:
>
> 2013-10-28 09:41:23,481 (SinkRunner-PollingRunner-**DefaultSinkProcessor)
> [ERROR - org.apache.flume.sink.hdfs.**HDFSEventSink.process(**HDFSEventSink.java:422)]
> process failed
> java.lang.VerifyError: class org.apache.hadoop.hdfs.**protocol.proto.**
> ClientNamenodeProtocolProtos$**SetOwnerRequestProto overrides final
> method getUnknownFields.()Lcom/**google/protobuf/**UnknownFieldSet;
>     at java.lang.ClassLoader.**defineClass1(Native Method)
>     at java.lang.ClassLoader.**defineClass(ClassLoader.java:**800)
>     at java.security.**SecureClassLoader.defineClass(**
> SecureClassLoader.java:142)
>     at java.net.URLClassLoader.**defineClass(URLClassLoader.**java:449)
>     at java.net.URLClassLoader.**access$100(URLClassLoader.**java:71)
>     at java.net.URLClassLoader$1.run(**URLClassLoader.java:361)
>     at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
>     at java.security.**AccessController.doPrivileged(**Native Method)
>     at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
>     at java.lang.ClassLoader.**loadClass(ClassLoader.java:**425)
>     at sun.misc.Launcher$**AppClassLoader.loadClass(**Launcher.java:308)
>     at java.lang.ClassLoader.**loadClass(ClassLoader.java:**358)
>     at java.lang.Class.**getDeclaredMethods0(Native Method)
>     at java.lang.Class.**privateGetDeclaredMethods(**Class.java:2531)
>     at java.lang.Class.**privateGetPublicMethods(Class.**java:2651)
>     at java.lang.Class.**privateGetPublicMethods(Class.**java:2661)
>     at java.lang.Class.getMethods(**Class.java:1467)
>
>  I am currently new to flume and any help shall be really appreciated.
>
> Regards,
>
> Wesley
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that