Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Different behaviour with same Pig Version


+
cscetbon.ext@... 2013-05-03, 09:44
+
cscetbon.ext@... 2013-05-06, 08:07
+
Prashant Kommireddi 2013-05-06, 17:35
Copy link to this message
-
Re: Different behaviour with same Pig Version
I think it's a bug in the default hadoop-env.sh :

# Extra Java runtime options.  Empty by default.
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_CLIENT_OPTS"

we should be able to overload this in the console without having to go into hadoop-env.sh to change it. So I replaced this line by :

# Extra Java runtime options.  Empty by default.
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_CLIENT_OPTS $HADOOP_OPTS"

Regards
--
Cyril SCETBON

On May 6, 2013, at 7:35 PM, Prashant Kommireddi <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:

This blog post describes some of this behavior
http://hadoopified.wordpress.com/2013/04/07/pig-startup-script-behavior/
On Mon, May 6, 2013 at 1:07 AM, <[EMAIL PROTECTED]> wrote:

forget it ! It came from hadoop-env.sh provided by the package I installed
on one cluster :(

Regards
--
Cyril SCETBON

On May 3, 2013, at 11:44 AM, [EMAIL PROTECTED]<mailto:
[EMAIL PROTECTED]> wrote:

Hi,

I have 2 differents behaviour for the same Pig version (with hadoop 1.1.2)
on different servers. If anyone can tell me why with the same versions and
the same parameters I don't have the same behaviour. I'm trying to set the
builtin import list but it's not used with one hadoop cluster whereas it's
used when I use it on a single node cluster :

+ PIG_OPTS=' -Dudf.import.list=org.apache.cassandra.hadoop.pig
-Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar
-Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs'
+ PIG_OPTS=' -Dudf.import.list=org.apache.cassandra.hadoop.pig
-Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar
-Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs
-Dpig.log.file=pig.log'
+ PIG_OPTS=' -Dudf.import.list=org.apache.cassandra.hadoop.pig
-Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar
-Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs
-Dpig.log.file=pig.log
-Dpig.home.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/..'
+ '[' '' == true ']'
+ '[' -n /usr/local/bin/hadoop ']'
+ '[' false == true ']'
+ '[' -f /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-withouthadoop.jar
']'
++ echo
/usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar
+
PIG_JAR=/usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar
+ '[' -z
/usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar ']'
+ '[' -n
/usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar ']'
+
CLASSPATH='/etc/hadoop/conf:/usr/lib/jvm/java-7-oracle/lib/tools.jar:::/opt/apache-cassandra-1.2.3/lib/*:/usr/share/hadoop/lib/*:/opt/pig-0.11.1/pig-0.11.1-withouthadoop.jar:/etc/hadoop:/opt/pig-0.11.1/bin/../lib/jython-standalone-2.5.3.jar:/opt/pig-0.11.1/bin/../pig-0.11.1-withouthadoop.jar'
+ export
'HADOOP_CLASSPATH=/etc/hadoop/conf:/usr/lib/jvm/java-7-oracle/lib/tools.jar:::/opt/apache-cassandra-1.2.3/lib/*:/usr/share/hadoop/lib/*:/opt/pig-0.11.1/pig-0.11.1-withouthadoop.jar:/etc/hadoop:/opt/pig-0.11.1/bin/../lib/jython-standalone-2.5.3.jar:/opt/pig-0.11.1/bin/../pig-0.11.1-withouthadoop.jar:'
+
HADOOP_CLASSPATH='/etc/hadoop/conf:/usr/lib/jvm/java-7-oracle/lib/tools.jar:::/opt/apache-cassandra-1.2.3/lib/*:/usr/share/hadoop/lib/*:/opt/pig-0.11.1/pig-0.11.1-withouthadoop.jar:/etc/hadoop:/opt/pig-0.11.1/bin/../lib/jython-standalone-2.5.3.jar:/opt/pig-0.11.1/bin/../pig-0.11.1-withouthadoop.jar:'
+ export 'HADOOP_OPTS=-Xmx1000m
-Dudf.import.list=org.apache.cassandra.hadoop.pig
-Dpig.additional.jars=/opt/apache-cassandra-1.2.3/lib/apache-cassandra-thrift-1.2.3.jar:/opt/apache-cassandra-1.2.3/lib/libthrift-0.7.0.jar
-4 log4j.properties -Dpig.log.dir=/opt/pig-0.11.1/bin/../logs
-Dpig.log.file=pig.log -Dpig.home.dir=/opt/pig-0.11.1/bin/..
-Djava.net.preferIPv4Stack=true '
+ HADOOP_OPTS='-Xmx1000m
-Dudf.import.list=org.apache.cassandra.hadoop.pig
-Dpig.additional.jars=/opt/apache-cassandra-1.2.3/lib/apache-cassandra-thrift-1.2.3.jar:/opt/apache-cassandra-1.2.3/lib/libthrift-0.7.0.jar
-4 log4j.properties -Dpig.log.dir=/opt/pig-0.11.1/bin/../logs
-Dpig.log.file=pig.log -Dpig.home.dir=/opt/pig-0.11.1/bin/..
-Djava.net.preferIPv4Stack=true '
+ '[' false == true ']'
+ exec /usr/bin/hadoop jar
/opt/pig-0.11.1/bin/../pig-0.11.1-withouthadoop.jar ./hadoop-sum-value_1.pig
2013-05-03 07:57:55,140 [main] INFO  org.apache.pig.Main - Apache Pig
version 0.11.1 (r1459641) compiled Mar 22 2013, 02:13:53
2013-05-03 07:57:55,141 [main] INFO  org.apache.pig.Main - Logging error
messages to: /home/ubuntu/pig_1367567875107.log
2013-05-03 07:57:57,502 [main] INFO  org.apache.pig.impl.util.Utils -
Default bootup file /home/ubuntu/.pigbootup not found
2013-05-03 07:57:58,461 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to hadoop file system at: hdfs://hadoop-cass-1:9100
2013-05-03 07:57:59,663 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to map-reduce job tracker at: hadoop-cass-1:8021
2013-05-03 07:58:01,387 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 1070: Could not resolve CassandraStorage2 using imports: [,
org.apache.pig.builtin., org.apache.pig.impl.builtin.]

+ PIG_OPTS=' -Dudf.import.list=org.apache.cassandra.hadoop.pig
-Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar
-Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs'
+ PIG_OPTS=' -Dudf.import.list=org.apache.cassandra.hadoop.pig
-Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar
-Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs
-Dpig.log.file=pig.log'
+ PIG_OPTS=' -Dudf.imp
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB