Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> RCfile


+
yingnan.ma 2012-05-24, 06:25
+
Harsh J 2012-05-24, 08:23
+
Raghu Angadi 2012-05-24, 18:22
Copy link to this message
-
Re: Re: RCfile
Hi,

the main reason which I want use the RCfile ,because the IO bottleneck is my big problem,and I want to try it, and test.
Moreover if I use the RCFilePigStorage, may I use the LZO compression together? or do you have some suggestion about improve the hadoop clustering performance.

Best Regards

Malone
2012-05-25

yingnan.ma

发件人: Raghu Angadi
发送时间: 2012-05-25  02:26:58
收件人: user
抄送:
主题: Re: RCfile
 
another option is
RCFilePigStorage.java<https://github.com/rangadi/elephant-bird/blob/rcfile_pig_storage/src/java/com/twitter/elephantbird/pig/store/RCFilePigStorage.java>
in
Elephantbird.
It is a drop-in replacement for default PigStorage and simple to use.
details on IO problem you want to fix?
On Thu, May 24, 2012 at 1:23 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> Malone,
>
> You should be able to follow this javadoc page to load Hive RCFiles in
> Pig:
> http://pig.apache.org/docs/r0.9.2/api/org/apache/pig/piggybank/storage/HiveColumnarLoader.html
>
> On Thu, May 24, 2012 at 11:55 AM, yingnan.ma <[EMAIL PROTECTED]>
> wrote:
> >
> > Hi,
> >
> > I want to use RCfile to address the IO problem, and I can not find some
> paper about how to install or how to use it
> > by PIG, so if you had some install or configue file, you could share
> with me. Thank you.
> >
> >
> >
> > Best Regards
> >
> > Malone
> >
> >
> > 2012-05-24
> >
> >
>
>
>
> --
> Harsh J
>
+
Raghu Angadi 2012-05-25, 14:45