yeah, you can enable lzo compression the normal way :
----
set mapred.output.compress true;
set mapred.output.compression.codec org.apache.hadoop.io.compress.LzoCodec;
store a into 'output' using
RCFilePigStorage();
----
Raghu.
On Thu, May 24, 2012 at 11:08 PM, yingnan.ma <[EMAIL PROTECTED]> wrote:
> Hi,
>
> the main reason which I want use the
RCfile ,because the IO bottleneck is
> my big problem,and I want to try it, and test.
> Moreover if I use the
RCFilePigStorage, may I use the LZO compression
> together? or do you have some suggestion about improve the hadoop
> clustering performance.
>
> Best Regards
>
> Malone
>
>
> 2012-05-25
>
>
>
> yingnan.ma
>
>
>
> 发件人： Raghu Angadi
> 发送时间： 2012-05-25 02:26:58
> 收件人： user
> 抄送：
> 主题： Re:
RCfile>
> another option is
>
RCFilePigStorage.java<
>
https://github.com/rangadi/elephant-bird/blob/rcfile_pig_storage/src/java/com/twitter/elephantbird/pig/store/RCFilePigStorage.java> >
> in
> Elephantbird.
> It is a drop-in replacement for default PigStorage and simple to use.
> details on IO problem you want to fix?
> On Thu, May 24, 2012 at 1:23 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> > Malone,
> >
> > You should be able to follow this javadoc page to load Hive
RCFiles in
> > Pig:
> >
>
http://pig.apache.org/docs/r0.9.2/api/org/apache/pig/piggybank/storage/HiveColumnarLoader.html> >
> > On Thu, May 24, 2012 at 11:55 AM, yingnan.ma <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > Hi,
> > >
> > > I want to use
RCfile to address the IO problem, and I can not find some
> > paper about how to install or how to use it
> > > by PIG, so if you had some install or configue file, you could share
> > with me. Thank you.
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Malone
> > >
> > >
> > > 2012-05-24
> > >
> > >
> >
> >
> >
> > --
> > Harsh J
> >
>
