Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume Agent not able to write to HDFS Sink

Copy link to this message
Re: Flume Agent not able to write to HDFS Sink

I did not answer your question fully.

Generally, this has something to do with the order in which entries are
listed in your classpath.

Make sure the version of the dependency the code is using is that one that
is in your class path first.

Take a look at the following documentation for additional information on
how classes are discovered in the classpath:


So lets say we have two jar files that containing the class

(1) a.jar contains the class org.apache.flume.formatter.output.BucketPath
with the method sanitizeString()

(2) b.jar contains the class org.apache.flume.formatter.output.BucketPath
with the method escapeString()

If your classpath is as follows:


And your code is trying to invoke
org.apache.flume.formatter.output.BucketPath.escapeString() you will have

This is because BucketPath was found first in a.jar but that definition
does not contain the method escapeString()

You will need to reorder the jar file entries in the class path so that the
correct version is picked up and used in your application.

On 24 April 2013 12:52, Israel Ekpo <[EMAIL PROTECTED]> wrote:

> Which version of Flume are you using?
> If this is something that just changed recently, consider moving the new
> jar to the end of your class path so that it does not override the
> important classes
> Shekhar Sharma <[EMAIL PROTECTED]> wrote:
> Can you tell me why this error comes up generally? I have checked the code
> and that method is present..
> i think it is looking out for method in some other jar ( may be older)
> which doesnt have that method..
> BTW it was working previously...but i have created my custom sink and i
> kept the jar in the lib folder of flume..and this creating problem...Will
> update if i find something..
> Regards,
> Som Shekhar Sharma
> +91-8197243810
> On Wed, Apr 24, 2013 at 9:04 PM, Israel Ekpo <[EMAIL PROTECTED]> wrote:
>> If you are just starting out, I think you should grab the latest version
>> of Flume (1.3.1)
>> http://flume.apache.org/download.html
>> Version 1.2.0 is an older version so working with the latest version is
>> better.
>> Try it out and send your feedback.
>> On 24 April 2013 10:15, Shekhar Sharma <[EMAIL PROTECTED]> wrote:
>>> Hello All,
>>> I am using a HDFS sink , but it is throwing the following error:
>>> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
>>> java.lang.NoSuchMethodError:
>>> org.apache.flume.formatter.output.BucketPath.escapeString(Ljava/lang/String;Ljava/util/Map;ZII)Ljava/lang/String;
>>>  at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:389)
>>> at
>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
>>>  at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
>>> at java.lang.Thread.run(Thread.java:722)
>>> I have seen the similar error
>>> https://issues.apache.org/jira/browse/FLUME-2019
>>> But i don't think this would be the error, since i have built the flume
>>> source code on my own system..so there shouldn't be any version mismatch
>>> The class path which it is taking while running it as follows:
>>> exec /usr/java/jdk1.7.0_10/bin/java -Xmx20m -cp
>>> '/home/training/flume-1.2.0-incubating-SNAPSHOT/conf:/home/training/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/hadoop/libexec/../conf:/usr/java/jdk1.7.0_10/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop/libexec/../lib/hsqldb-