Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Can't use snappy codec


Copy link to this message
-
Re: Can't use snappy codec
Patrick Wendell 2012-08-03, 18:55
Oh I see - so the current version "just works" for you. I
misinterpreted your last email thinking you had done something
customized to fix it.

The script must have been fixed in an earlier JIRA. Thanks for your
thorough explanation.

- Patrick

On Fri, Aug 3, 2012 at 11:21 AM, Eran Kutner <[EMAIL PROTECTED]> wrote:
> Hi Patrick,
>
> With the new script it's working ok.
> The old script (the one that comes with cdh4) had a problem
>
> This code in the old script:
>     local HADOOP_JAVA_LIBRARY_PATH=$(HADOOP_CLASSPATH="$FLUME_CLASSPATH" \
>         ${HADOOP_IN_PATH} org.apache.flume.tools.GetJavaProperty \
>         java.library.path 2>/dev/null)
>
> Would set the HADOOP_JAVA_LIBRARY_PATH to
> "java.library.path=//usr/lib/hadoop/lib/native"
> Which would end up setting FLUME_JAVA_LIBRARY_PATH to
> ":java.library.path=//usr/lib/hadoop/lib/native"
> That would then be used to start the processes as
> "-Djava.library.path=:java.library.path=//usr/lib/hadoop/lib/native" which
> is obviously wrong.
>
> The new script has code to clean up the extra "java.library.path" returned
> by the above code.
> So there are two options, either the implementation of
> org.apache.flume.tools.GetJavaProperty changed, so it now returns the extra
> parameter name and therefore it was required to remove it in the script, or
> the original one included in CDH4 had a bug that was fixed in 1.2
>
> -eran
>
>
>
> On Fri, Aug 3, 2012 at 7:43 PM, Patrick Wendell <[EMAIL PROTECTED]> wrote:
>>
>> Hey Eran,
>>
>> So the flume-ng script works by trying to figure out what library path
>> Hadoop is using and then replicating that for flume. If
>> HADOOP_JAVA_LIBRARY_PATH is set it will try to use that. Otherwise it
>> tries to infer the path based on what the hadoop script itself
>> determines.
>>
>> What is the path getting set to in your case and how does that differ
>> from expecations? Just trying to figure out what the bug is.
>>
>> - Patrick
>>
>> On Fri, Aug 3, 2012 at 9:25 AM, Eran Kutner <[EMAIL PROTECTED]> wrote:
>> > Thanks Patrick, that helped me figure out the problem and it looks like
>> > a
>> > bug in the "flume-ng" file provided with CDH4, it was messing the
>> > library.path.
>> > I copied the file that was included in flume 1.2.0 distribution and it
>> > now
>> > works ok.
>> >
>> > Thanks for your help.
>> >
>> > -eran
>> >
>> >
>> >
>> > On Fri, Aug 3, 2012 at 6:36 PM, Patrick Wendell <[EMAIL PROTECTED]>
>> > wrote:
>> >>
>> >> Hey Eran,
>> >>
>> >> You need to make sure the Flume JVM gets passed
>> >> -Djava.library.path=XXX with the correct path to where your native
>> >> snappy libraries are located.
>> >>
>> >> You can set this by adding the option directly to the flume-ng runner
>> >> script.
>> >>
>> >> - Patrick
>> >>
>> >> On Fri, Aug 3, 2012 at 7:33 AM, Eran Kutner <[EMAIL PROTECTED]> wrote:
>> >> > Hi,
>> >> > I'm trying to use the snappy codec but keep getting "native snappy
>> >> > library
>> >> > not available" errors.
>> >> > I'm using CDH4 but replaced the flume 1.1 JARs that are included with
>> >> > that
>> >> > distribution with flume 1.2 JARs.
>> >> > I tried anything I can think of, including symlinking the hadoop
>> >> > native
>> >> > library under flume-ng/lib/ dirctory both nothing helps.
>> >> > Any idea how to resolve this?
>> >> >
>> >> > This is the error:
>> >> > 2012-08-03 10:23:30,598 WARN util.NativeCodeLoader: Unable to load
>> >> > native-hadoop library for your platform... using builtin-java classes
>> >> > where
>> >> > applicable
>> >> > 2012-08-03 10:23:35,670 WARN hdfs.HDFSEventSink: HDFS IO error
>> >> > java.io.IOException: java.lang.RuntimeException: native snappy
>> >> > library
>> >> > not
>> >> > available
>> >> >         at
>> >> > org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:202)
>> >> >         at
>> >> >
>> >> > org.apache.flume.sink.hdfs.BucketWriter.access$000(BucketWriter.java:48)
>> >> >         at
>> >> > org.apache.flume.sink.hdfs.BucketWriter$1.run(BucketWriter.java:155)
>>