Hi,

I want to use RCfile to address the IO problem, and I can not find some paper about how to install or how to use it
by PIG, so if you had some install or configue file, you could share with me. Thank you.

Best Regards

Malone
2012-05-24

  

Malone,

You should be able to follow this javadoc page to load Hive RCFiles in
Pig: http://pig.apache.org/docs/r0.9.2/api/org/apache/pig/piggybank/storage/HiveColumnarLoader.html

On Thu, May 24, 2012 at 11:55 AM, yingnan.ma <[EMAIL PROTECTED]> wrote:
>
> Hi,
>
> I want to use RCfile to address the IO problem, and I can not find some paper about how to install or how to use it
> by PIG, so if you had some install or configue file, you could share with me. Thank you.
>
>
>
> Best Regards
>
> Malone
>
>
> 2012-05-24
>
>

--
Harsh J

another option is
RCFilePigStorage.java<https://github.com/rangadi/elephant-bird/blob/rcfile_pig_storage/src/java/com/twitter/elephantbird/pig/store/RCFilePigStorage.java>
in
Elephantbird.
It is a drop-in replacement for default PigStorage and simple to use.

details on IO problem you want to fix?

On Thu, May 24, 2012 at 1:23 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> Malone,
>
> You should be able to follow this javadoc page to load Hive RCFiles in
> Pig:
> http://pig.apache.org/docs/r0.9.2/api/org/apache/pig/piggybank/storage/HiveColumnarLoader.html
>
> On Thu, May 24, 2012 at 11:55 AM, yingnan.ma <[EMAIL PROTECTED]>
> wrote:
> >
> > Hi,
> >
> > I want to use RCfile to address the IO problem, and I can not find some
> paper about how to install or how to use it
> > by PIG, so if you had some install or configue file, you could share
> with me. Thank you.
> >
> >
> >
> > Best Regards
> >
> > Malone
> >
> >
> > 2012-05-24
> >
> >
>
>
>
> --
> Harsh J
>

Hi,

the main reason which I want use the RCfile ,because the IO bottleneck is my big problem,and I want to try it, and test.
Moreover if I use the RCFilePigStorage, may I use the LZO compression together? or do you have some suggestion about improve the hadoop clustering performance.

Best Regards

Malone
2012-05-25

yingnan.ma

发件人： Raghu Angadi
发送时间： 2012-05-25  02:26:58
收件人： user
抄送：
主题： Re: RCfile
 
another option is
RCFilePigStorage.java<https://github.com/rangadi/elephant-bird/blob/rcfile_pig_storage/src/java/com/twitter/elephantbird/pig/store/RCFilePigStorage.java>
in
Elephantbird.
It is a drop-in replacement for default PigStorage and simple to use.
details on IO problem you want to fix?
On Thu, May 24, 2012 at 1:23 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> Malone,
>
> You should be able to follow this javadoc page to load Hive RCFiles in
> Pig:
> http://pig.apache.org/docs/r0.9.2/api/org/apache/pig/piggybank/storage/HiveColumnarLoader.html
>
> On Thu, May 24, 2012 at 11:55 AM, yingnan.ma <[EMAIL PROTECTED]>
> wrote:
> >
> > Hi,
> >
> > I want to use RCfile to address the IO problem, and I can not find some
> paper about how to install or how to use it
> > by PIG, so if you had some install or configue file, you could share
> with me. Thank you.
> >
> >
> >
> > Best Regards
> >
> > Malone
> >
> >
> > 2012-05-24
> >
> >
>
>
>
> --
> Harsh J
>

yeah, you can enable lzo compression the normal way :

----
set mapred.output.compress true;
set mapred.output.compression.codec org.apache.hadoop.io.compress.LzoCodec;

store a into 'output' using RCFilePigStorage();
----

Raghu.

On Thu, May 24, 2012 at 11:08 PM, yingnan.ma <[EMAIL PROTECTED]> wrote:

> Hi,
>
> the main reason which I want use the RCfile ,because the IO bottleneck is
> my big problem,and I want to try it, and test.
> Moreover if I use the RCFilePigStorage, may I use the LZO compression
> together? or do you have some suggestion about improve the hadoop
> clustering performance.
>
> Best Regards
>
> Malone
>
>
> 2012-05-25
>
>
>
> yingnan.ma
>
>
>
> 发件人： Raghu Angadi
> 发送时间： 2012-05-25 02:26:58
> 收件人： user
> 抄送：
> 主题： Re: RCfile
>
> another option is
> RCFilePigStorage.java<
> https://github.com/rangadi/elephant-bird/blob/rcfile_pig_storage/src/java/com/twitter/elephantbird/pig/store/RCFilePigStorage.java
> >
> in
> Elephantbird.
> It is a drop-in replacement for default PigStorage and simple to use.
> details on IO problem you want to fix?
> On Thu, May 24, 2012 at 1:23 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> > Malone,
> >
> > You should be able to follow this javadoc page to load Hive RCFiles in
> > Pig:
> >
> http://pig.apache.org/docs/r0.9.2/api/org/apache/pig/piggybank/storage/HiveColumnarLoader.html
> >
> > On Thu, May 24, 2012 at 11:55 AM, yingnan.ma <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > Hi,
> > >
> > > I want to use RCfile to address the IO problem, and I can not find some
> > paper about how to install or how to use it
> > > by PIG, so if you had some install or configue file, you could share
> > with me. Thank you.
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Malone
> > >
> > >
> > > 2012-05-24
> > >
> > >
> >
> >
> >
> > --
> > Harsh J
> >
>