Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - FileChannel on Windows


Copy link to this message
-
Re: FileChannel on Windows
Brock Noland 2013-04-10, 21:32
http://issues.apache.org/jira/browse/FLUME-1285
On Wed, Apr 10, 2013 at 4:20 PM, Israel Ekpo <[EMAIL PROTECTED]> wrote:

> Brock, do you have the JIRA issue where the patches were uploaded?
>
> If I get some spare time this weekend I can take a shot at it.
>
>
>
> On 10 April 2013 16:44, Brock Noland <[EMAIL PROTECTED]> wrote:
>
>> I think we just need to copy 4-5 classes over from hadoop. Someone
>> uploaded a patch I just haven't had time to work with it.
>>
>>
>> On Wed, Apr 10, 2013 at 3:41 PM, Hari Shreedharan <
>> [EMAIL PROTECTED]> wrote:
>>
>>>  Yes. We will need to remove the format upgrade code and the old format
>>> code from the file channel as well. I think it should be possible to do it,
>>> but I suspect it is just that no one had the time to look at this as yet.
>>>
>>> --
>>> Hari Shreedharan
>>>
>>> On Wednesday, April 10, 2013 at 1:38 PM, Roshan Naik wrote:
>>>
>>> Hari,
>>>   i recall some discussion around the desire to get rid of this
>>> dependency. i guess it means removing that piece of code that supports the
>>> format upgrade ? are there any plans to do that ?
>>>  -roshan
>>>
>>>
>>> On Wed, Apr 10, 2013 at 1:25 PM, Paul Chavez <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>> **
>>> I did not realize I could use pre-built binaries on Windows. Thank you,
>>> I used this to grab the hadoop-common.jar from our existing Hadoop cluster
>>> and there are no startup errors now.
>>>
>>> Thanks to you, Brock and Hari.
>>>
>>> -Paul
>>>
>>>  ------------------------------
>>> *From:* Israel Ekpo [mailto:[EMAIL PROTECTED]]
>>> *Sent:* Wednesday, April 10, 2013 12:07 PM
>>> *To:* [EMAIL PROTECTED]
>>> *Subject:* Re: FileChannel on Windows
>>>
>>> Paul,
>>>
>>> If Flume 1.3.1 is what you are looking for, you don't have to build it
>>> from source.
>>>
>>> You can just download it directly from the site. Its already released.
>>>
>>> Paul Chavez <[EMAIL PROTECTED]> wrote:
>>>
>>> Hello,
>>>
>>> I've been using flume on Windows to send logs to an avro source for
>>> awhile now, using the AvroClient class. I'm now trying to get a full agent
>>> running to hopefully leverage the spoolDir and httpSource functionality.
>>> Part of my requirements is a persistent channel so I'm attempting to get a
>>> config running the utilizes the fileChannel.
>>>
>>> I followed the instructions at
>>> http://mapredit.blogspot.com/2012/07/run-flume-13x-on-windows.html to
>>> build flume, same as I did for the AvroClient. However it seems I need some
>>> HDFS libraries. Can anyone point me to instructions for building the JAR(s)
>>> I need to get this working?
>>>
>>> Currently I'm running into the following exception:
>>> 10 Apr 2013 11:50:29,317 INFO  [lifecycleSupervisor-1-0]
>>> (org.apache.flume.channel.file.FileChannel.start:285)  - Starting
>>> FileChannel fc_Default { dataDirs: [c:\flume_data\log] }...
>>> 10 Apr 2013 11:50:29,317 ERROR [lifecycleSupervisor-1-0]
>>> (org.apache.flume.channel.file.FileChannel.start:314)  - Failed to start
>>> the file channel [channel=fc_Default]
>>> java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
>>>  at java.lang.ClassLoader.defineClass1(Native Method)
>>>  at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
>>>  at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
>>>  at
>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>>>  at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>>>  at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>>>  at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>>>  at java.security.AccessController.doPrivileged(Native Method)
>>>  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>  at org.apache.flume.channel.file.Log$Builder.build(Log.java:212)
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org