Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # dev >> Why does Pig not use default resources from the Configuration object?


+
Bhooshan Mogal 2013-03-29, 01:10
Copy link to this message
-
Re: Why does Pig not use default resources from the Configuration object?
Hi Bhooshan,

There is a patch that addresses what you need, and is part of 0.12
(unreleased). Take a look and see if you can apply the patch to the version
you are using.
https://issues.apache.org/jira/browse/PIG-3135.

With this patch, the following property will allow you to override the
default and pass in your own configuration.
pig.use.overriden.hadoop.configs=true
On Thu, Mar 28, 2013 at 6:10 PM, Bhooshan Mogal <[EMAIL PROTECTED]>wrote:

> Hi Folks,
>
> I had implemented the Hadoop FileSystem abstract class for a storage system
> at work. This implementation uses some config files that are similar in
> structure to hadoop config files. They have a *-default.xml and a
> *-site.xml for users to override default properties. In the class that
> implemented the Hadoop FileSystem, I had added these configuration files as
> default resources in a static block using
> Configuration.addDefaultResource("my-default.xml") and
> Configuration.addDefaultResource("my-site.xml". This was working fine and
> we were able to run the Hadoop Filesystem CLI and map-reduce jobs just fine
> for our storage system. However, when we tried using this storage system in
> pig scripts, we saw errors indicating that our configuration parameters
> were not available. Upon further debugging, we saw that the config files
> were added to the Configuration object as resources, but were part of
> defaultResources. However, in Main.java in the pig source, we saw that the
> Configuration object was created as Configuration conf = new
> Configuration(false);, thereby setting loadDefaults to false in the conf
> object. As a result, properties from the default resources (including my
> config files) were not loaded and hence, unavailable.
>
> We solved the problem by using Configuration.addResource instead of
> Configuration.addDefaultResource, but still could not figure out why Pig
> does not use default resources?
>
> Could someone on the list explain why this is the case?
>
> Thanks,
> --
> Bhooshan
>