Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Why does Pig not use default resources from the Configuration object?


Copy link to this message
-
Re: Why does Pig not use default resources from the Configuration object?
Hi Bhooshan,

There is a patch that addresses what you need, and is part of 0.12
(unreleased). Take a look and see if you can apply the patch to the version
you are using.
https://issues.apache.org/jira/browse/PIG-3135.

With this patch, the following property will allow you to override the
default and pass in your own configuration.
pig.use.overriden.hadoop.configs=true
On Thu, Mar 28, 2013 at 6:10 PM, Bhooshan Mogal <[EMAIL PROTECTED]>wrote:

> Hi Folks,
>
> I had implemented the Hadoop FileSystem abstract class for a storage system
> at work. This implementation uses some config files that are similar in
> structure to hadoop config files. They have a *-default.xml and a
> *-site.xml for users to override default properties. In the class that
> implemented the Hadoop FileSystem, I had added these configuration files as
> default resources in a static block using
> Configuration.addDefaultResource("my-default.xml") and
> Configuration.addDefaultResource("my-site.xml". This was working fine and
> we were able to run the Hadoop Filesystem CLI and map-reduce jobs just fine
> for our storage system. However, when we tried using this storage system in
> pig scripts, we saw errors indicating that our configuration parameters
> were not available. Upon further debugging, we saw that the config files
> were added to the Configuration object as resources, but were part of
> defaultResources. However, in Main.java in the pig source, we saw that the
> Configuration object was created as Configuration conf = new
> Configuration(false);, thereby setting loadDefaults to false in the conf
> object. As a result, properties from the default resources (including my
> config files) were not loaded and hence, unavailable.
>
> We solved the problem by using Configuration.addResource instead of
> Configuration.addDefaultResource, but still could not figure out why Pig
> does not use default resources?
>
> Could someone on the list explain why this is the case?
>
> Thanks,
> --
> Bhooshan
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB