Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> RCfile


Copy link to this message
-
Re: Re: RCfile
Hi,

the main reason which I want use the RCfile ,because the IO bottleneck is my big problem,and I want to try it, and test.
Moreover if I use the RCFilePigStorage, may I use the LZO compression together? or do you have some suggestion about improve the hadoop clustering performance.

Best Regards

Malone
2012-05-25

yingnan.ma

发件人: Raghu Angadi
发送时间: 2012-05-25  02:26:58
收件人: user
抄送:
主题: Re: RCfile
 
another option is
RCFilePigStorage.java<https://github.com/rangadi/elephant-bird/blob/rcfile_pig_storage/src/java/com/twitter/elephantbird/pig/store/RCFilePigStorage.java>
in
Elephantbird.
It is a drop-in replacement for default PigStorage and simple to use.
details on IO problem you want to fix?
On Thu, May 24, 2012 at 1:23 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> Malone,
>
> You should be able to follow this javadoc page to load Hive RCFiles in
> Pig:
> http://pig.apache.org/docs/r0.9.2/api/org/apache/pig/piggybank/storage/HiveColumnarLoader.html
>
> On Thu, May 24, 2012 at 11:55 AM, yingnan.ma <[EMAIL PROTECTED]>
> wrote:
> >
> > Hi,
> >
> > I want to use RCfile to address the IO problem, and I can not find some
> paper about how to install or how to use it
> > by PIG, so if you had some install or configue file, you could share
> with me. Thank you.
> >
> >
> >
> > Best Regards
> >
> > Malone
> >
> >
> > 2012-05-24
> >
> >
>
>
>
> --
> Harsh J
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB