Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - FileChannel on Windows


Copy link to this message
-
Re: FileChannel on Windows
Hari Shreedharan 2013-04-10, 20:58
Ah, yes, I remember that patch. It basically copied the Writables interfaces/classes I think. Yeah, copying those would be a nice workaround (I think those dont have any implementation anyway).
Hari

--
Hari Shreedharan
On Wednesday, April 10, 2013 at 1:44 PM, Brock Noland wrote:

> I think we just need to copy 4-5 classes over from hadoop. Someone uploaded a patch I just haven't had time to work with it.
>
>
> On Wed, Apr 10, 2013 at 3:41 PM, Hari Shreedharan <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > Yes. We will need to remove the format upgrade code and the old format code from the file channel as well. I think it should be possible to do it, but I suspect it is just that no one had the time to look at this as yet.
> >
> > --
> > Hari Shreedharan
> >
> >
> > On Wednesday, April 10, 2013 at 1:38 PM, Roshan Naik wrote:
> >
> > > Hari,
> > >   i recall some discussion around the desire to get rid of this dependency. i guess it means removing that piece of code that supports the format upgrade ? are there any plans to do that ?
> > > -roshan
> > >
> > >
> > >
> > > On Wed, Apr 10, 2013 at 1:25 PM, Paul Chavez <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > > I did not realize I could use pre-built binaries on Windows. Thank you, I used this to grab the hadoop-common.jar from our existing Hadoop cluster and there are no startup errors now.
> > > >  
> > > > Thanks to you, Brock and Hari.
> > > >  
> > > > -Paul
> > > >
> > > > From: Israel Ekpo [mailto:[EMAIL PROTECTED]]
> > > > Sent: Wednesday, April 10, 2013 12:07 PM
> > > > To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> > > > Subject: Re: FileChannel on Windows
> > > >
> > > > Paul,
> > > >
> > > > If Flume 1.3.1 is what you are looking for, you don't have to build it from source.
> > > >
> > > > You can just download it directly from the site. Its already released.
> > > >
> > > > Paul Chavez <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > >
> > > > Hello,
> > > >  
> > > > I've been using flume on Windows to send logs to an avro source for awhile now, using the AvroClient class. I'm now trying to get a full agent running to hopefully leverage the spoolDir and httpSource functionality. Part of my requirements is a persistent channel so I'm attempting to get a config running the utilizes the fileChannel.
> > > >  
> > > > I followed the instructions at http://mapredit.blogspot.com/2012/07/run-flume-13x-on-windows.html to build flume, same as I did for the AvroClient. However it seems I need some HDFS libraries. Can anyone point me to instructions for building the JAR(s) I need to get this working?
> > > >  
> > > > Currently I'm running into the following exception:
> > > > 10 Apr 2013 11:50:29,317 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.FileChannel.start:285)  - Starting FileChannel fc_Default { dataDirs: [c:\flume_data\log] }...
> > > > 10 Apr 2013 11:50:29,317 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.FileChannel.start:314)  - Failed to start the file channel [channel=fc_Default]
> > > > java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
> > > >  at java.lang.ClassLoader.defineClass1(Native Method)
> > > >  at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
> > > >  at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
> > > >  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> > > >  at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> > > >  at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> > > >  at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> > > >  at java.security.AccessController.doPrivileged(Native Method)
> > > >  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > > >  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > > >  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)