Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # dev - check scriptPid in ScriptOperator.java to avoid BrokenPipeException: script_broken_pipe1.q, script_broken_pipe2.q, minimr_broken_pipe.q won't FAILED


Copy link to this message
-
check scriptPid in ScriptOperator.java to avoid BrokenPipeException: script_broken_pipe1.q, script_broken_pipe2.q, minimr_broken_pipe.q won't FAILED
Bing Li 2012-02-20, 09:57
Hi, developers
I found that in ScriptOperator.close(), when it catch an IOException, it
doesn't check the status of the scriptPid, I mean if the process is null.
which may cause a race condition.
I did a change that

In ScriptOperator.close()
try {
  if ( scriptOutWriter != null ) {
    scriptOutWriter.close();
  }
} catch (IOException e ) {
  if ( (isBrokenPipeException(e) && allowPartialConsumption()) *||
(scriptPid != null && scriptPid.exitValue() == 0)*) {
   ...
  }
Also the related logic in ScriptOperator.processOp().
After that, I found the following UT cases failed, because got a successful
job instead of a failure:
    [junit] Tests run: 332, Failures: 2, Errors: 0, Time elapsed: 836.821
sec
    [junit] Test org.apache.hadoop.hive.cli.TestNegativeCliDriver FAILED
    script_broken_pipe1.q
    script_broken_pipe2.q

    [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 40.389 sec
    [junit] Test org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver
FAILED
    minimr_broken_pipe.q
Is it acceptable?
Thanks,
- Bing