Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - pig not respecting parameters in mapred-site.xml


+
Suhas Satish 2013-08-06, 20:16
Copy link to this message
-
Re: pig not respecting parameters in mapred-site.xml
Prashant Kommireddi 2013-08-06, 20:32
Can you tell us how exactly you are running the pig script? Is your
mapred-site.xml on the classpath? Are you trying to run this via grunt or
Java APIs?
On Tue, Aug 6, 2013 at 1:16 PM, Suhas Satish <[EMAIL PROTECTED]> wrote:

> I am running pig on a custom hadoop implementation but it doesnt respect
> params in mapred-site.xml.
>
> Looking into the code, I find that the following 2 files are slightly
> different from stock hadoop in that some patches are not present.
>
> hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
> and
> src/mapred/org/apache/hadoop/mapred/JobConf.java
>
> Given the constraint that I cannot modify these files, what change should I
> make within pig to recognize mapred-site.xml parameters?
>
> I pulled in PIG-3135 and PIG-3145 which make changes to
>  org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java
>
> But the params in mapred-site.xml are still not getting recognized. Upon
> remote eclipse  debugging with breakpoints in the file above, this is what
> I found -
>
> HExecutionEngine.java  - jc = new  jobConf()
> calls
> 1st call upon JobConf() constructor -
> Configuration.get(String)
>  Configuration.getProps() --> if properties ==null, properties = new
> Properties(); loadResources(properties, resources...);
>
> JobConf static constructor -
> Configuration.addDefaultResource(mapred-site.xml)
>
> HExecutionEngine: 2nd call jc.addResource("mapred-site.xml") -
> Configuration.get
> val = getProps().getProperty(name)
>  if (val==null) val > <custom_hadoop_impl_configuration_object>.getDefault(name);
>
> HExecutionEngine: 3rd call   recomputeProperties(jc, properties) -->
> clearing properties which were added so it gets reloaded again.
>
>
> What do I ned to do to make sure the  getProps().getProperty call is not
> null  so that the mapred-site.xml values are not over-ridden by defaults in
> custom_hadoop implementation ?
>
> Thanks,
> Suhas.
>
+
Suhas Satish 2013-08-07, 00:32
+
Suhas Satish 2013-08-07, 02:44
+
Prashant Kommireddi 2013-08-07, 07:18
+
Serega Sheypak 2013-08-06, 20:28