Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Can't use snappy codec


+
Eran Kutner 2012-08-03, 14:33
Copy link to this message
-
Re: Can't use snappy codec
Hey Eran,

You need to make sure the Flume JVM gets passed
-Djava.library.path=XXX with the correct path to where your native
snappy libraries are located.

You can set this by adding the option directly to the flume-ng runner script.

- Patrick

On Fri, Aug 3, 2012 at 7:33 AM, Eran Kutner <[EMAIL PROTECTED]> wrote:
> Hi,
> I'm trying to use the snappy codec but keep getting "native snappy library
> not available" errors.
> I'm using CDH4 but replaced the flume 1.1 JARs that are included with that
> distribution with flume 1.2 JARs.
> I tried anything I can think of, including symlinking the hadoop native
> library under flume-ng/lib/ dirctory both nothing helps.
> Any idea how to resolve this?
>
> This is the error:
> 2012-08-03 10:23:30,598 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2012-08-03 10:23:35,670 WARN hdfs.HDFSEventSink: HDFS IO error
> java.io.IOException: java.lang.RuntimeException: native snappy library not
> available
>         at
> org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:202)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.access$000(BucketWriter.java:48)
>         at
> org.apache.flume.sink.hdfs.BucketWriter$1.run(BucketWriter.java:155)
>         at
> org.apache.flume.sink.hdfs.BucketWriter$1.run(BucketWriter.java:152)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:125)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:152)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:307)
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:717)
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:714)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.RuntimeException: native snappy library not available
>         at
> org.apache.hadoop.io.compress.SnappyCodec.createCompressor(SnappyCodec.java:135)
>         at
> org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:84)
>         at
> org.apache.flume.sink.hdfs.HDFSCompressedDataStream.open(HDFSCompressedDataStream.java:70)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:195)
>         ... 13 more
>
> And my sink configuration:
> flume05.sinks.hdfsSink.type = hdfs
> #flume05.sinks.hdfsSink.type = logger
> flume05.sinks.hdfsSink.channel = memoryChannel
> flume05.sinks.hdfsSink.hdfs.path=hdfs://hadoop2-m1:8020/test-events/%Y-%m-%d
> flume05.sinks.hdfsSink.hdfs.filePrefix=raw-events.avro
> flume05.sinks.hdfsSink.hdfs.rollInterval=60
> flume05.sinks.hdfsSink.hdfs.rollCount=0
> flume05.sinks.hdfsSink.hdfs.rollSize=0
> flume05.sinks.hdfsSink.hdfs.fileType=CompressedStream
> flume05.sinks.hdfsSink.hdfs.codeC=snappy
> flume05.sinks.hdfsSink.hdfs.writeFormat=Text
> flume05.sinks.hdfsSink.hdfs.batchSize=1000
> flume05.sinks.hdfsSink.serializer = avro_event
>
> Thanks.
>
> -eran
>
+
Eran Kutner 2012-08-03, 16:25
+
Patrick Wendell 2012-08-03, 16:43
+
Eran Kutner 2012-08-03, 18:21
+
Patrick Wendell 2012-08-03, 18:55
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB