Hi

I have downloaded and deployed latest flume code from repository
"https://svn.apache.org/repos/asf/flume/trunk/"

In the conf file I am using following properties for the hdfs sink:
agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.hdfs.path= <xyz>
agent.sinks.hdfsSink.hdfs.fileType =CompressedStream
agent.sinks.hdfsSink.hdfs.filePrefix=<xxx>
agent.sinks.hdfsSink.hdfs.rollSize=0
agent.sinks.hdfsSink.hdfs.codeC=bzip2
agent.sinks.hdfsSink.hdfs.rollCount=0
agent.sinks.hdfsSink.hdfs.batchSize=1
agent.sinks.hdfsSink.hdfs.rollInterval=1

After running it gives me an exception for bzip.

java.lang.IllegalArgumentException: Unsupported compression codec bzip2.
  Please choose from: [None, GzipCodec, DefaultCodec, LzoCodec,
LzopCodec, SnappyCodec]

Isn't flume 1.2.0 supporting bzip2?

Regards,
Jagadish

Hi Jagadish,
I believe that flume is using hadoop framework to get list of compression codecs. Is bzip2 codec available in your hadoop environment?

You might want to check configuration property io.compression.codecs for inclusion of org.apache.hadoop.io.compress.BZip2Codec.

Jarcec

On Jul 25, 2012, at 12:20 PM, Jagadish Bihani wrote:

> Hi
>
> I have downloaded and deployed latest flume code from repository
> "https://svn.apache.org/repos/asf/flume/trunk/"
>
> In the conf file I am using following properties for the hdfs sink:
>
>
> agent.sinks.hdfsSink.type = hdfs
> agent.sinks.hdfsSink.hdfs.path= <xyz>
> agent.sinks.hdfsSink.hdfs.fileType =CompressedStream
> agent.sinks.hdfsSink.hdfs.filePrefix=<xxx>
> agent.sinks.hdfsSink.hdfs.rollSize=0
> agent.sinks.hdfsSink.hdfs.codeC=bzip2
> agent.sinks.hdfsSink.hdfs.rollCount=0
> agent.sinks.hdfsSink.hdfs.batchSize=1
> agent.sinks.hdfsSink.hdfs.rollInterval=1
>
> After running it gives me an exception for bzip.
>
> java.lang.IllegalArgumentException: Unsupported compression codec bzip2.  Please choose from: [None, GzipCodec, DefaultCodec, LzoCodec, LzopCodec, SnappyCodec]
>
> Isn't flume 1.2.0 supporting bzip2?
>
> Regards,
> Jagadish

Thanks Jarek.
Indeed it was missing from config file. Adding it did the trick.

On 07/25/2012 08:24 PM, Jarek Jarcec Cecho wrote:
> Hi Jagadish,
> I believe that flume is using hadoop framework to get list of compression codecs. Is bzip2 codec available in your hadoop environment?
>
> You might want to check configuration property io.compression.codecs for inclusion of org.apache.hadoop.io.compress.BZip2Codec.
>
> Jarcec
>
> On Jul 25, 2012, at 12:20 PM, Jagadish Bihani wrote:
>
>> Hi
>>
>> I have downloaded and deployed latest flume code from repository
>> "https://svn.apache.org/repos/asf/flume/trunk/"
>>
>> In the conf file I am using following properties for the hdfs sink:
>>
>>
>> agent.sinks.hdfsSink.type = hdfs
>> agent.sinks.hdfsSink.hdfs.path= <xyz>
>> agent.sinks.hdfsSink.hdfs.fileType =CompressedStream
>> agent.sinks.hdfsSink.hdfs.filePrefix=<xxx>
>> agent.sinks.hdfsSink.hdfs.rollSize=0
>> agent.sinks.hdfsSink.hdfs.codeC=bzip2
>> agent.sinks.hdfsSink.hdfs.rollCount=0
>> agent.sinks.hdfsSink.hdfs.batchSize=1
>> agent.sinks.hdfsSink.hdfs.rollInterval=1
>>
>> After running it gives me an exception for bzip.
>>
>> java.lang.IllegalArgumentException: Unsupported compression codec bzip2.  Please choose from: [None, GzipCodec, DefaultCodec, LzoCodec, LzopCodec, SnappyCodec]
>>
>> Isn't flume 1.2.0 supporting bzip2?
>>
>> Regards,
>> Jagadish