Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> FLumeNG -HDFS errors


Copy link to this message
-
Re: FLumeNG -HDFS errors
HI,

Your Hadoop libraries are not in classpath, from where you run Flume.

The following should work: Please change the paths to your respective
installations,

1)

C:\Users>set CLASSPATH=.;D:\Source\my-flume\my-flume;D:\Source\apache-
flume-1.3.0-SNAPSHOT-dist\apache-flume-1.3.0-SNAPSHOT\lib\*;D:\Source\hadoop-1.0
.3\hadoop-1.0.3\*;D:\Source\hadoop-1.0.3\hadoop-1.0.3\lib\*

2)

C:\Users>java org.apache.flume.node.Application -f D:\Source\my-flume\
my-flume\flume-conf.properties -n host1
Best Regards,
On Fri, Oct 12, 2012 at 3:41 PM, Rewti Ingle
<[EMAIL PROTECTED]>wrote:

>   Hi, ** ****
>
> ** **
>
> I have installed FlumeNG [1.2] and Hadoop [1.0.3].****
>
> ** **
>
> I am getting following error  while running flume agent with HDFS sink.***
> *
>
> ** **
>
> 2012-10-11 16:55:57,640 WARN hdfs.HDFSEventSink: HDFS IO error****
>
> java.io.IOException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException:
> org.apache.hadoop.hdfs.DistributedFileSystem****
>
>                 at
> org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:202)****
>
>                 at
> org.apache.flume.sink.hdfs.BucketWriter.access$000(BucketWriter.java:48)**
> **
>
>                 at
> org.apache.flume.sink.hdfs.BucketWriter$1.run(BucketWriter.java:155)****
>
>                 at
> org.apache.flume.sink.hdfs.BucketWriter$1.run(BucketWriter.java:152)****
>
>                 at
> org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:125)
> ****
>
>                 at
> org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:152)****
>
>                 at
> org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:307)****
>
>                 at
> org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:717)***
> *
>
>                 at
> org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:714)***
> *
>
>                 at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)****
>
>                 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> ****
>
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> ****
>
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> ****
>
>                 at java.lang.Thread.run(Thread.java:662)****
>
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.hadoop.hdfs.DistributedFileSystem****
>
>                 at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1128)****
>
>                 at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1729)****
>
>                 at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:74)****
>
>                 at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1768)***
> *
>
>                 at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1750)****
>
>                 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:234)
> ****
>
>                 at org.apache.hadoop.fs.Path.getFileSystem(Path.java:189)*
> ***
>
>                 at
> org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:186)****
>
>                 ... 13 more****
>
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hdfs.DistributedFileSystem****
>
>                 at java.net.URLClassLoader$1.run(URLClassLoader.java:202)*
> ***
>
>                 at java.security.AccessController.doPrivileged(Native
> Method)****
>
>                 at
> java.net.URLClassLoader.findClass(URLClassLoader.java:190)****
>
>                 at java.lang.ClassLoader.loadClass(ClassLoader.java:306)**
> **
>
>                 at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)****
>
>                 at java.lang.ClassLoader.loadClass(ClassLoader.java:247)**
> **
>
>                 at java.lang.Class.forName0(Native Method)****
>
>                 at java.lang.Class.forName(Class.java:247)****