Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - RCfile


+
yingnan.ma 2012-05-24, 06:25
+
Harsh J 2012-05-24, 08:23
+
Raghu Angadi 2012-05-24, 18:22
+
yingnan.ma 2012-05-25, 06:08
Copy link to this message
-
Re: Re: RCfile
Raghu Angadi 2012-05-25, 14:45
yeah, you can enable lzo compression the normal way :

----
set mapred.output.compress true;
set mapred.output.compression.codec org.apache.hadoop.io.compress.LzoCodec;

store a into 'output' using RCFilePigStorage();
----

Raghu.

On Thu, May 24, 2012 at 11:08 PM, yingnan.ma <[EMAIL PROTECTED]> wrote:

> Hi,
>
> the main reason which I want use the RCfile ,because the IO bottleneck is
> my big problem,and I want to try it, and test.
> Moreover if I use the RCFilePigStorage, may I use the LZO compression
> together? or do you have some suggestion about improve the hadoop
> clustering performance.
>
> Best Regards
>
> Malone
>
>
> 2012-05-25
>
>
>
> yingnan.ma
>
>
>
> 发件人: Raghu Angadi
> 发送时间: 2012-05-25 02:26:58
> 收件人: user
> 抄送:
> 主题: Re: RCfile
>
> another option is
> RCFilePigStorage.java<
> https://github.com/rangadi/elephant-bird/blob/rcfile_pig_storage/src/java/com/twitter/elephantbird/pig/store/RCFilePigStorage.java
> >
> in
> Elephantbird.
> It is a drop-in replacement for default PigStorage and simple to use.
> details on IO problem you want to fix?
> On Thu, May 24, 2012 at 1:23 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> > Malone,
> >
> > You should be able to follow this javadoc page to load Hive RCFiles in
> > Pig:
> >
> http://pig.apache.org/docs/r0.9.2/api/org/apache/pig/piggybank/storage/HiveColumnarLoader.html
> >
> > On Thu, May 24, 2012 at 11:55 AM, yingnan.ma <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > Hi,
> > >
> > > I want to use RCfile to address the IO problem, and I can not find some
> > paper about how to install or how to use it
> > > by PIG, so if you had some install or configue file, you could share
> > with me. Thank you.
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Malone
> > >
> > >
> > > 2012-05-24
> > >
> > >
> >
> >
> >
> > --
> > Harsh J
> >
>