Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # dev >> Review Request: SQOOP-674 Sqoop2: Exceptions in special map reduce threads can cause mapreduce job to freeze


Copy link to this message
-
Re: Review Request: SQOOP-674 Sqoop2: Exceptions in special map reduce threads can cause mapreduce job to freeze


> On Dec. 16, 2012, 10:12 p.m., Hari Shreedharan wrote:
> > Jarcec,
> >
> > Excellent catch! Running tryAcquire in a loop is generally an expensive CPU consuming operation. You could do something like this:
> >
> > * Release the free semaphore after setting readerFinished to true, just before throwing the exception in the consumer whenever exceptions are thrown (2 places I think).
> > * Exchange the order of calls to free.acquire and checkIfConsumerThrew (acquire the free sema before checking if the consumer threw).
> >
> > This would make sure that we don't try to acquire the semaphore in a loop, but yet ensure that we will release the semaphore when an exception is thrown so we can continue and throw the exception out

Also I'd recommend adding a unit test. It should be fairly easy to write a dummy loader that throws after a fixed number of operations. Such a test would fail without this patch.
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8622/#review14569
-----------------------------------------------------------
On Dec. 16, 2012, 3:22 a.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8622/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2012, 3:22 a.m.)
>
>
> Review request for Sqoop and Hari Shreedharan.
>
>
> Description
> -------
>
> I've changed unconditional free.acquire() to do-while loop that will periodically check reader status.
>
>
> This addresses bug SQOOP-674.
>     https://issues.apache.org/jira/browse/SQOOP-674
>
>
> Diffs
> -----
>
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 38e2292781b10bab577f3aa8af47d2d0c9365e46
>
> Diff: https://reviews.apache.org/r/8622/diff/
>
>
> Testing
> -------
>
> Unit tests are passing and I've tested it on real cluster.
>
>
> Thanks,
>
> Jarek Cecho
>
>