Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> RCfile


+
yingnan.ma 2012-05-24, 06:25
+
Harsh J 2012-05-24, 08:23
+
Raghu Angadi 2012-05-24, 18:22
+
yingnan.ma 2012-05-25, 06:08
Copy link to this message
-
Re: Re: RCfile
yeah, you can enable lzo compression the normal way :

----
set mapred.output.compress true;
set mapred.output.compression.codec org.apache.hadoop.io.compress.LzoCodec;

store a into 'output' using RCFilePigStorage();
----

Raghu.

On Thu, May 24, 2012 at 11:08 PM, yingnan.ma <[EMAIL PROTECTED]> wrote:

> Hi,
>
> the main reason which I want use the RCfile ,because the IO bottleneck is
> my big problem,and I want to try it, and test.
> Moreover if I use the RCFilePigStorage, may I use the LZO compression
> together? or do you have some suggestion about improve the hadoop
> clustering performance.
>
> Best Regards
>
> Malone
>
>
> 2012-05-25
>
>
>
> yingnan.ma
>
>
>
> 发件人: Raghu Angadi
> 发送时间: 2012-05-25 02:26:58
> 收件人: user
> 抄送:
> 主题: Re: RCfile
>
> another option is
> RCFilePigStorage.java<
> https://github.com/rangadi/elephant-bird/blob/rcfile_pig_storage/src/java/com/twitter/elephantbird/pig/store/RCFilePigStorage.java
> >
> in
> Elephantbird.
> It is a drop-in replacement for default PigStorage and simple to use.
> details on IO problem you want to fix?
> On Thu, May 24, 2012 at 1:23 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> > Malone,
> >
> > You should be able to follow this javadoc page to load Hive RCFiles in
> > Pig:
> >
> http://pig.apache.org/docs/r0.9.2/api/org/apache/pig/piggybank/storage/HiveColumnarLoader.html
> >
> > On Thu, May 24, 2012 at 11:55 AM, yingnan.ma <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > Hi,
> > >
> > > I want to use RCfile to address the IO problem, and I can not find some
> > paper about how to install or how to use it
> > > by PIG, so if you had some install or configue file, you could share
> > with me. Thank you.
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Malone
> > >
> > >
> > > 2012-05-24
> > >
> > >
> >
> >
> >
> > --
> > Harsh J
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB