Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Question on custom store function


Copy link to this message
-
Re: Question on custom store function
Hey Felix,

>> The only problem is that in the setStoreLocation function we have to call
>> FileOutputFormat.setOutputPath(job, new Path(location));

Cant you massage location to appropriate string you want to?

Ashutosh

On Tue, Nov 1, 2011 at 18:07, felix gao <[EMAIL PROTECTED]> wrote:

> I have wrote a custom store function that primarily based on the
> multi-storage store function.  They way I use it is
>
>
> store load_log INTO
> '/Users/felix/Documents/pig/multi_store_output/ns_{0}/site_{1}' using
> MyMultiStorage('2,1', '1,2');
> where {0} and {1} will be substituted with the tuple index at 0 and index
> at 1.  Everything is fine and all the data is written to the correct place.
>  The only problem is that in the setStoreLocation function we have to call
> FileOutputFormat.setOutputPath(job, new Path(location)); i have
> 'Users/felix/Documents/pig/multi_store_output/ns_{0}/site_{1}' as my output
> location so there is actually a folder created in my fs with ns_{0}
> and site_{1}.  Is there a way to tell hadoop not to create those output
> directory?
>
> Thanks,
>
> Felix
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB