Hello

I am trying to write compressed files in HDFS. If I am using the Flume CompressedStream type with HdfsSink, it is working fine.
But If I am writing code outside and running, it is giving null pointer exception. I must be missing something. Any help?
If I change to simple file to sequence file it is working. If I do not have compression then also it is working.

Error:
java.lang.NullPointerException
        at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:64)
        at org.apache.hadoop.io.compress.GzipCodec.createOutputStream(GzipCodec.java:97)
        at cieg.atap.DiSnAbs.open(DiSnAbs.java:52)
        at cieg.atap.DiAgManager$DiAgWorker.run(DiAgManager.java:205)

SampleCode:
        InputStream in = new FileInputStream(new File(inFile));
        OutputStream out = fileSystem.create(outFile);
        if (System.getProperty("hdfs.compression.type") != null) {
            CompressionCodec lCodec = getCodec(System.getProperty("hdfs.compression.type"));
            out = lCodec.createOutputStream(out);
        }
    public static CompressionCodec getCompressionCodec(String value)
    {  
        if (value.equals("gzip"))
            return new GzipCodec();
        else if (value.equals("bzip2"))
            return new BZip2Codec();
        else
            return new SnappyCodec();
    }

Thanks
Mallikharjuna Rao
Director, Software Engineering
603-791-8604 (W) 603 296 7721 (C)

Hi Mallik,

On Tue, Mar 19, 2013 at 2:11 PM, Rao, Mallik <[EMAIL PROTECTED]>wrote:

> But If I am writing code outside and running, it is giving null pointer
> exception.
What do you mean writing code outside? I'm not sure I understand what
you're doing here. Is this still within Flume?

Regards,
Mike

Goal is to put inside as a custom sink. But for now testing outside FLUME.

Thanks
Mallikharjuna Rao
Director, Software Engineering
603-791-8604 (W) 603 296 7721 (C)

-----Original Message-----
From: Mike Percy [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, March 19, 2013 7:05 PM
To: [EMAIL PROTECTED]
Subject: Re: Compressed Files NPE

Hi Mallik,

On Tue, Mar 19, 2013 at 2:11 PM, Rao, Mallik <[EMAIL PROTECTED]>wrote:

> But If I am writing code outside and running, it is giving null pointer
> exception.
What do you mean writing code outside? I'm not sure I understand what
you're doing here. Is this still within Flume?

Regards,
Mike

Hadoop is open source so you could look @ the line that is throwing... but I'm guessing your classpath or library path is missing something.

Regards,
Mike

Sent from my iPhone

On Mar 20, 2013, at 5:06 AM, "Rao, Mallik" <[EMAIL PROTECTED]> wrote:

> Goal is to put inside as a custom sink. But for now testing outside FLUME.
>
> Thanks
> Mallikharjuna Rao
> Director, Software Engineering
> 603-791-8604 (W) 603 296 7721 (C)
>
> -----Original Message-----
> From: Mike Percy [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 19, 2013 7:05 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Compressed Files NPE
>
> Hi Mallik,
>
> On Tue, Mar 19, 2013 at 2:11 PM, Rao, Mallik <[EMAIL PROTECTED]>wrote:
>
>> But If I am writing code outside and running, it is giving null pointer
>> exception.
>
>
> What do you mean writing code outside? I'm not sure I understand what
> you're doing here. Is this still within Flume?
>
> Regards,
> Mike