Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> FileChannel on Windows


+
Paul Chavez 2013-04-10, 18:55
+
Brock Noland 2013-04-10, 19:07
+
Hari Shreedharan 2013-04-10, 19:08
+
Israel Ekpo 2013-04-10, 20:22
+
Israel Ekpo 2013-04-10, 19:06
+
Paul Chavez 2013-04-10, 20:25
+
Roshan Naik 2013-04-10, 20:38
+
Hari Shreedharan 2013-04-10, 20:41
Copy link to this message
-
Re: FileChannel on Windows
I think we just need to copy 4-5 classes over from hadoop. Someone uploaded
a patch I just haven't had time to work with it.
On Wed, Apr 10, 2013 at 3:41 PM, Hari Shreedharan <[EMAIL PROTECTED]
> wrote:

>  Yes. We will need to remove the format upgrade code and the old format
> code from the file channel as well. I think it should be possible to do it,
> but I suspect it is just that no one had the time to look at this as yet.
>
> --
> Hari Shreedharan
>
> On Wednesday, April 10, 2013 at 1:38 PM, Roshan Naik wrote:
>
> Hari,
>   i recall some discussion around the desire to get rid of this
> dependency. i guess it means removing that piece of code that supports the
> format upgrade ? are there any plans to do that ?
>  -roshan
>
>
> On Wed, Apr 10, 2013 at 1:25 PM, Paul Chavez <
> [EMAIL PROTECTED]> wrote:
>
> **
> I did not realize I could use pre-built binaries on Windows. Thank you, I
> used this to grab the hadoop-common.jar from our existing Hadoop cluster
> and there are no startup errors now.
>
> Thanks to you, Brock and Hari.
>
> -Paul
>
>  ------------------------------
> *From:* Israel Ekpo [mailto:[EMAIL PROTECTED]]
> *Sent:* Wednesday, April 10, 2013 12:07 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: FileChannel on Windows
>
> Paul,
>
> If Flume 1.3.1 is what you are looking for, you don't have to build it
> from source.
>
> You can just download it directly from the site. Its already released.
>
> Paul Chavez <[EMAIL PROTECTED]> wrote:
>
> Hello,
>
> I've been using flume on Windows to send logs to an avro source for awhile
> now, using the AvroClient class. I'm now trying to get a full agent running
> to hopefully leverage the spoolDir and httpSource functionality. Part of my
> requirements is a persistent channel so I'm attempting to get a config
> running the utilizes the fileChannel.
>
> I followed the instructions at
> http://mapredit.blogspot.com/2012/07/run-flume-13x-on-windows.html to
> build flume, same as I did for the AvroClient. However it seems I need some
> HDFS libraries. Can anyone point me to instructions for building the JAR(s)
> I need to get this working?
>
> Currently I'm running into the following exception:
> 10 Apr 2013 11:50:29,317 INFO  [lifecycleSupervisor-1-0]
> (org.apache.flume.channel.file.FileChannel.start:285)  - Starting
> FileChannel fc_Default { dataDirs: [c:\flume_data\log] }...
> 10 Apr 2013 11:50:29,317 ERROR [lifecycleSupervisor-1-0]
> (org.apache.flume.channel.file.FileChannel.start:314)  - Failed to start
> the file channel [channel=fc_Default]
> java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
>  at java.lang.ClassLoader.defineClass1(Native Method)
>  at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
>  at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
>  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>  at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>  at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>  at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>  at org.apache.flume.channel.file.Log$Builder.build(Log.java:212)
>  at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
>  at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
>  at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
+
Hari Shreedharan 2013-04-10, 20:58
+
Israel Ekpo 2013-04-10, 21:20
+
Brock Noland 2013-04-10, 21:32
+
Israel Ekpo 2013-04-10, 21:45
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB