Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # dev >> Possible issue when using non-bundled hadoop


Copy link to this message
-
Possible issue when using non-bundled hadoop
A pig user brought to my attention that PIG_OPTS are not being picked up
but "pig -Dfoo=bar script.pig" works. I dug a little into it and found that
in the case when an external hadoop (not bundled with pig) is used we rely
on bin/hadoop script for starting up pig. In the process we pass PIG_OPTS
as HADOOP_OPTS to the hadoop script.

bin/hadoop in turn picks up HADOOP_OPTS from
HADOOP_HOME/conf/hadoop-env.sh. It might be possible a user completely
resets HADOOP_OPTS here and PIG_OPTS is cleared.

# Extra Java runtime options.  Empty by default.
export HADOOP_OPTS="-server"

This will be noticeable since Pig 0.9.1
https://issues.apache.org/jira/browse/PIG-2239

Users need to be aware of this and make sure they do not reset HADOOP_OPTS.
Instead this will work
# Extra Java runtime options.  Empty by default.
export HADOOP_OPTS="-server $HADOOP_OPTS"

Thought it would be useful info if this ever comes up.

-Prashant
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB