Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # dev >> Review Request: SQOOP-674 Sqoop2: Exceptions in special map reduce threads can cause mapreduce job to freeze


Copy link to this message
-
Re: Review Request: SQOOP-674 Sqoop2: Exceptions in special map reduce threads can cause mapreduce job to freeze

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8622/#review14569
-----------------------------------------------------------
Jarcec,

Excellent catch! Running tryAcquire in a loop is generally an expensive CPU consuming operation. You could do something like this:

* Release the free semaphore after setting readerFinished to true, just before throwing the exception in the consumer whenever exceptions are thrown (2 places I think).
* Exchange the order of calls to free.acquire and checkIfConsumerThrew (acquire the free sema before checking if the consumer threw).

This would make sure that we don't try to acquire the semaphore in a loop, but yet ensure that we will release the semaphore when an exception is thrown so we can continue and throw the exception out

- Hari Shreedharan
On Dec. 16, 2012, 3:22 a.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8622/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2012, 3:22 a.m.)
>
>
> Review request for Sqoop and Hari Shreedharan.
>
>
> Description
> -------
>
> I've changed unconditional free.acquire() to do-while loop that will periodically check reader status.
>
>
> This addresses bug SQOOP-674.
>     https://issues.apache.org/jira/browse/SQOOP-674
>
>
> Diffs
> -----
>
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 38e2292781b10bab577f3aa8af47d2d0c9365e46
>
> Diff: https://reviews.apache.org/r/8622/diff/
>
>
> Testing
> -------
>
> Unit tests are passing and I've tested it on real cluster.
>
>
> Thanks,
>
> Jarek Cecho
>
>