Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> pig not respecting parameters in mapred-site.xml


+
Suhas Satish 2013-08-06, 20:16
Copy link to this message
-
Re: pig not respecting parameters in mapred-site.xml
Can you tell us how exactly you are running the pig script? Is your
mapred-site.xml on the classpath? Are you trying to run this via grunt or
Java APIs?
On Tue, Aug 6, 2013 at 1:16 PM, Suhas Satish <[EMAIL PROTECTED]> wrote:

> I am running pig on a custom hadoop implementation but it doesnt respect
> params in mapred-site.xml.
>
> Looking into the code, I find that the following 2 files are slightly
> different from stock hadoop in that some patches are not present.
>
> hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
> and
> src/mapred/org/apache/hadoop/mapred/JobConf.java
>
> Given the constraint that I cannot modify these files, what change should I
> make within pig to recognize mapred-site.xml parameters?
>
> I pulled in PIG-3135 and PIG-3145 which make changes to
>  org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java
>
> But the params in mapred-site.xml are still not getting recognized. Upon
> remote eclipse  debugging with breakpoints in the file above, this is what
> I found -
>
> HExecutionEngine.java  - jc = new  jobConf()
> calls
> 1st call upon JobConf() constructor -
> Configuration.get(String)
>  Configuration.getProps() --> if properties ==null, properties = new
> Properties(); loadResources(properties, resources...);
>
> JobConf static constructor -
> Configuration.addDefaultResource(mapred-site.xml)
>
> HExecutionEngine: 2nd call jc.addResource("mapred-site.xml") -
> Configuration.get
> val = getProps().getProperty(name)
>  if (val==null) val > <custom_hadoop_impl_configuration_object>.getDefault(name);
>
> HExecutionEngine: 3rd call   recomputeProperties(jc, properties) -->
> clearing properties which were added so it gets reloaded again.
>
>
> What do I ned to do to make sure the  getProps().getProperty call is not
> null  so that the mapred-site.xml values are not over-ridden by defaults in
> custom_hadoop implementation ?
>
> Thanks,
> Suhas.
>
+
Suhas Satish 2013-08-07, 00:32
+
Suhas Satish 2013-08-07, 02:44
+
Prashant Kommireddi 2013-08-07, 07:18
+
Serega Sheypak 2013-08-06, 20:28
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB