Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Can't use snappy codec


+
Eran Kutner 2012-08-03, 14:33
Copy link to this message
-
Re: Can't use snappy codec
Patrick Wendell 2012-08-03, 15:36
Hey Eran,

You need to make sure the Flume JVM gets passed
-Djava.library.path=XXX with the correct path to where your native
snappy libraries are located.

You can set this by adding the option directly to the flume-ng runner script.

- Patrick

On Fri, Aug 3, 2012 at 7:33 AM, Eran Kutner <[EMAIL PROTECTED]> wrote:
> Hi,
> I'm trying to use the snappy codec but keep getting "native snappy library
> not available" errors.
> I'm using CDH4 but replaced the flume 1.1 JARs that are included with that
> distribution with flume 1.2 JARs.
> I tried anything I can think of, including symlinking the hadoop native
> library under flume-ng/lib/ dirctory both nothing helps.
> Any idea how to resolve this?
>
> This is the error:
> 2012-08-03 10:23:30,598 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2012-08-03 10:23:35,670 WARN hdfs.HDFSEventSink: HDFS IO error
> java.io.IOException: java.lang.RuntimeException: native snappy library not
> available
>         at
> org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:202)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.access$000(BucketWriter.java:48)
>         at
> org.apache.flume.sink.hdfs.BucketWriter$1.run(BucketWriter.java:155)
>         at
> org.apache.flume.sink.hdfs.BucketWriter$1.run(BucketWriter.java:152)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:125)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:152)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:307)
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:717)
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:714)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.RuntimeException: native snappy library not available
>         at
> org.apache.hadoop.io.compress.SnappyCodec.createCompressor(SnappyCodec.java:135)
>         at
> org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:84)
>         at
> org.apache.flume.sink.hdfs.HDFSCompressedDataStream.open(HDFSCompressedDataStream.java:70)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:195)
>         ... 13 more
>
> And my sink configuration:
> flume05.sinks.hdfsSink.type = hdfs
> #flume05.sinks.hdfsSink.type = logger
> flume05.sinks.hdfsSink.channel = memoryChannel
> flume05.sinks.hdfsSink.hdfs.path=hdfs://hadoop2-m1:8020/test-events/%Y-%m-%d
> flume05.sinks.hdfsSink.hdfs.filePrefix=raw-events.avro
> flume05.sinks.hdfsSink.hdfs.rollInterval=60
> flume05.sinks.hdfsSink.hdfs.rollCount=0
> flume05.sinks.hdfsSink.hdfs.rollSize=0
> flume05.sinks.hdfsSink.hdfs.fileType=CompressedStream
> flume05.sinks.hdfsSink.hdfs.codeC=snappy
> flume05.sinks.hdfsSink.hdfs.writeFormat=Text
> flume05.sinks.hdfsSink.hdfs.batchSize=1000
> flume05.sinks.hdfsSink.serializer = avro_event
>
> Thanks.
>
> -eran
>
+
Eran Kutner 2012-08-03, 16:25
+
Patrick Wendell 2012-08-03, 16:43
+
Eran Kutner 2012-08-03, 18:21
+
Patrick Wendell 2012-08-03, 18:55