When running a test that fails, I can get the output diff and the hive log, but when the failure occurs in the query execution, I don't see how I can retrieve whatever was logged by the execution.
For example, I can see in the target\qfile-results\...q.out: PREHOOK: query: SELECT cint, ... AND (ctinyint > cbigint))) PREHOOK: type: QUERY PREHOOK: Input: default@alltypesorc PREHOOK: Output: file:/Users/rremus/Development/hive/itests/qtest/target/tmp/localscratchdir/hive_2014-02-06_12-17-05_641_8095220136958689426-1/-mr-10000 Execution failed with exit status: 2 Obtaining error information
Task failed! Task ID: Stage-1
/Users/rremus/Development/hive/itests/qtest/target/tmp/log/hive.log FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
The hive.log is a bit more detailed: 2014-02-06 12:17:05,741 INFO ql.Driver (SessionState.java:printInfo(504)) - Launching Job 1 out of 1 2014-02-06 12:17:05,742 INFO exec.Task (SessionState.java:printInfo(504)) - Number of reduce tasks is set to 0 since there's no reduce operator 2014-02-06 12:17:05,822 INFO mr.ExecDriver (MapRedTask.java:execute(175)) - Generating plan file file:/Users/rremus/Development/hive/itests/qtest/target/tmp/localscratchdir/hive_2014-02-06_12-17-05_641_8095220136958689426-1/-local-10003/plan.xml 2014-02-06 12:17:05,822 INFO log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities> 2014-02-06 12:17:05,823 INFO exec.Utilities (Utilities.java:serializePlan(783)) - Serializing MapredWork via kryo 2014-02-06 12:17:05,835 INFO log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=serializePlan start=1391717825822 end=1391717825835 duration=13 from=org.apache.hadoop.hive.ql.exec.Utilities> 2014-02-06 12:17:05,836 INFO mr.ExecDriver (MapRedTask.java:execute(207)) - Executing: /Users/rremus/Development/hive/itests/qtest/../../testutils/hadoop jar /Users/rremus/.m2/repository/org/apache/hive/hive-exec/0.13.0-SNAPSHOT/hive-exec-0.13.0-SNAPSHOT.jar org.apache.hadoop.hive.ql.exec.mr.ExecDriver -plan file:/Users/rremus/Development/hive/itests/qtest/target/tmp/localscratchdir/hive_2014-02-06_12-17-05_641_8095220136958689426-1/-local-10003/plan.xml -jobconffile file:/Users/rremus/Development/hive/itests/qtest/target/tmp/localscratchdir/hive_2014-02-06_12-17-05_641_8095220136958689426-1/-local-10002/jobconf.xml 2014-02-06 12:17:15,431 ERROR exec.Task (SessionState.java:printError(513)) - Execution failed with exit status: 2 2014-02-06 12:17:15,432 ERROR exec.Task (SessionState.java:printError(513)) - Obtaining error information 2014-02-06 12:17:15,432 ERROR exec.Task (SessionState.java:printError(513)) - Task failed! Task ID: Stage-1
Still, at the end of running the test there is nothing left in target/tmp/localscratchdir, scratchdir or in target/tmp/hadoop-tmp. In this case, I'm missing the output from the MR job that failed. Is there a way for me to get this missing output?
Re: Any way to preserve the .q unit tests MR execution output?
but no files in them. I'll try to dig and see whether the files get deleted or not created to start with...
thanks, ~Remus ________________________________________ From: Brock Noland <[EMAIL PROTECTED]> Sent: Thursday, February 6, 2014 11:58 PM To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: Any way to preserve the .q unit tests MR execution output?
Not sure about saving the output of the jobs, but I believe the mr logs should be in target/ somewhere or in /tmp/$USER
On Thu, Feb 6, 2014 at 3:34 PM, Remus Rusanu <[EMAIL PROTECTED]> wrote:
Thanks again, but no luck for me. I started debugging this and my suspicion is it that is related to the fact that I do not have any Hadoop deployed on this laptop. When testutil/hadoop is run there is $HADOP_HOME to look at, no proper hadoop-env.sh or anything alike. I think this results in log4j not being properly initialized for the "MR" task. I'll post back if I ever get to the end of it.
________________________________________ From: Brock Noland <[EMAIL PROTECTED]> Sent: Saturday, February 8, 2014 7:30 PM To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: Any way to preserve the .q unit tests MR execution output?
I just ran a test on my local machine and the MR logs went to /tmp/brock.
On Sat, Feb 8, 2014 at 6:12 AM, Remus Rusanu <[EMAIL PROTECTED]> wrote:
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext