Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # dev >> check scriptPid in ScriptOperator.java to avoid BrokenPipeException: script_broken_pipe1.q, script_broken_pipe2.q, minimr_broken_pipe.q won't FAILED


Copy link to this message
-
check scriptPid in ScriptOperator.java to avoid BrokenPipeException: script_broken_pipe1.q, script_broken_pipe2.q, minimr_broken_pipe.q won't FAILED
Hi, developers
I found that in ScriptOperator.close(), when it catch an IOException, it
doesn't check the status of the scriptPid, I mean if the process is null.
which may cause a race condition.
I did a change that

In ScriptOperator.close()
try {
  if ( scriptOutWriter != null ) {
    scriptOutWriter.close();
  }
} catch (IOException e ) {
  if ( (isBrokenPipeException(e) && allowPartialConsumption()) *||
(scriptPid != null && scriptPid.exitValue() == 0)*) {
   ...
  }
Also the related logic in ScriptOperator.processOp().
After that, I found the following UT cases failed, because got a successful
job instead of a failure:
    [junit] Tests run: 332, Failures: 2, Errors: 0, Time elapsed: 836.821
sec
    [junit] Test org.apache.hadoop.hive.cli.TestNegativeCliDriver FAILED
    script_broken_pipe1.q
    script_broken_pipe2.q

    [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 40.389 sec
    [junit] Test org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver
FAILED
    minimr_broken_pipe.q
Is it acceptable?
Thanks,
- Bing
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB