Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Writing to rcfile


+
Mohit Anchlia 2012-05-22, 21:14
Copy link to this message
-
Re: Writing to rcfile
Elephantbird has support RCFile storage. current version supports storing
Thrift and Protobufs.

You can try prototype implementation
RCFilePigStorage.java<https://github.com/rangadi/elephant-bird/blob/rcfile_pig_storage/src/java/com/twitter/elephantbird/pig/store/RCFilePigStorage.java>,
it can be used in place of PigStorage(). This is not yet committed, will
likely be included in next release (
https://github.com/kevinweil/elephant-bird/pull/206)

usage :
  register 'libs/*hive-common*.jar; -- hive-common for RCFile
  register 'libs/*hive-exec*.jar;  -- hive-exec for RCFile
  register 'libs/*protobuf-java*.jar; -- protobuf-java should not be
required, but for now it is.

  a = load 'x.txt' as (a, b);
  store a into 'output' using RCFilePigStorage();

On Tue, May 22, 2012 at 2:14 PM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:

> Could someone please point me to someway where I can store in rcfile format
> with snappy compression? I need to use this output in hive.
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB