Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill >> mail # dev >> Review Request 15564: Patch for DRILL-274


Copy link to this message
-
Re: Review Request 15564: Patch for DRILL-274
See inline.
On Fri, Nov 15, 2013 at 10:21 PM, Timothy Chen <[EMAIL PROTECTED]> wrote:

> Sorry got no internet access on my laptop so have to put comments here in
> email:
>
> 1, from the rawspoolbuffer class I see it's intended for multithreaded
> with volatiles, but I have question when multiple threads get from the
> buffer. Looks like when the buffer is empty and is unfinished, it blocks
> until it finds an item or a interrupt exception happens. I wonder if it is
> marked finished while it's blocked? Does it hang forever since there is no
> timeout on the take?
>
>

The SpoolingRawBatchBuffer is an implementation of the RawBatchBuffer
interface. For a while now, Drill has been using UnlimitedRawBatchBuffer.
In both implementations, this is handled by interrupting the thread and
catching the exception.

> 2, the spooling test doesn't seem to test spooling as I think it will pass
> even without spooling given enough direct memory right? Perhaps should
> verify the spooled data is same as expected?
I have a special configuration file that I use for this test which limits
the number of execution threads to 1. The physical plan has exchanges, so
that ensures that there are multiple fragments, and since there is only one
thread, it forces it to spool.
>
Tim
>
> Sent from my iPhone
>
> > On Nov 14, 2013, at 6:00 PM, "Steven Phillips" <[EMAIL PROTECTED]>
> wrote:
> >
> >
> > -----------------------------------------------------------
> > This is an automatically generated e-mail. To reply, visit:
> > https://reviews.apache.org/r/15564/
> > -----------------------------------------------------------
> >
> > Review request for drill.
> >
> >
> > Bugs: DRILL-274
> >    https://issues.apache.org/jira/browse/DRILL-274
> >
> >
> > Repository: drill-git
> >
> >
> > Description
> > -------
> >
> > DRILL-274: Spooling batch buffer
> >
> >
> > Diffs
> > -----
> >
> >  distribution/src/resources/drill-override.conf
> 7694cedf62ae5dde1221d3bef0ac6cdaedaf78d4
> >  exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
> 36504f6df36d3c3bf7bf6b158f777777125fef74
> >
>  exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java
> a7f6d2ef03eb2fef20fed5f327a12ea3c82989de
> >
>  exec/java-exec/src/main/java/org/apache/drill/exec/store/LocalSyncableFileSystem.java
> PRE-CREATION
> >
>  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/AbstractFragmentCollector.java
> d58de2fc1b7a3369cd904433ecd87f3c7d1019ec
> >
>  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/BitComHandlerImpl.java
> 5639851f4c99406f210b1d65c1c8ae3739f9dcfc
> >
>  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/IncomingBuffers.java
> c9e56085aeebfeb44b2ae82c981251a7d3e47675
> >
>  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/MergingCollector.java
> 670347cba07b4c5010053a89a6352973d40a5dca
> >
>  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/PartitionedCollector.java
> af127781bb1d6bfab312d2843a3d41eadbf9f4f4
> >
>  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/SpoolingRawBatchBuffer.java
> PRE-CREATION
> >
>  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/UnlimitedRawBatchBuffer.java
> 43870da545a4659f85bf608c4d2fda1f808765cf
> >
>  exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/RunningFragmentManager.java
> 588316b3e5e3f0d9f35d6fb30070001ddb539f4e
> >
>  exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/RemoteFragmentHandler.java
> d947d683d4686cabf30b85b927ed8fe3847ef6f1
> >  exec/java-exec/src/main/resources/drill-module.conf
> 725c6b4d41bb0b24876e74e7aed7a83f5e6f42f5
> >
>  exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestRepeatedFunction.java
> ac05319725e0bf797a67a133f44a9a6988ab12c2
> >
>  exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestComparisonFunctions.java
> c47fa89ab65b18ca21fd4b2f2dc9d478b5fe4809
> >
>  exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestOptiqPlans.java