Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Drill >> mail # dev >> Re: [5/8] git commit: DRILL-176: Updates to affinity calculator, fixes for parquet serialization. Fix to ErrorHelper looping


Copy link to this message
-
Re: [5/8] git commit: DRILL-176: Updates to affinity calculator, fixes for parquet serialization. Fix to ErrorHelper looping
Hi,

when i try mvn install after these changes the
org.apache.drill.exec.store.parquet.ParquetRecordReaderTest is hanging.

Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.596 sec -
in org.apache.drill.exec.expr.ExpressionTest
Running org.apache.drill.exec.store.TestAffinityCalculator
Took 0.616287 ms to build range map
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.137 sec -
in org.apache.drill.exec.store.TestAffinityCalculator
Running org.apache.drill.exec.store.parquet.ParquetRecordReaderTest
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.

Environment is Fedora 18, open jdk 1.7

with skipTests everything is getting compiled fine.

Regards
Tanujit

On Fri, Aug 23, 2013 at 5:36 AM, <[EMAIL PROTECTED]> wrote:

> DRILL-176:  Updates to affinity calculator, fixes for parquet
> serialization.  Fix to ErrorHelper looping
>
>
> Project: http://git-wip-us.apache.org/repos/asf/incubator-drill/repo
> Commit:
> http://git-wip-us.apache.org/repos/asf/incubator-drill/commit/7edd3617
> Tree: http://git-wip-us.apache.org/repos/asf/incubator-drill/tree/7edd3617
> Diff: http://git-wip-us.apache.org/repos/asf/incubator-drill/diff/7edd3617
>
> Branch: refs/heads/master
> Commit: 7edd36170a9be291a69e44f6090474193485bf14
> Parents: d6ae53e
> Author: Steven Phillips <[EMAIL PROTECTED]>
> Authored: Thu Aug 22 16:18:55 2013 -0700
> Committer: Jacques Nadeau <[EMAIL PROTECTED]>
> Committed: Thu Aug 22 16:18:55 2013 -0700
>
> ----------------------------------------------------------------------
>  .../drill/exec/planner/fragment/Wrapper.java    |   5 +-
>  .../drill/exec/store/AffinityCalculator.java    |  91 ++++++----
>  .../exec/store/parquet/ParquetGroupScan.java    | 177 +++++++++----------
>  .../exec/store/parquet/ParquetRecordReader.java |   2 +-
>  .../store/parquet/ParquetScanBatchCreator.java  |  10 +-
>  .../drill/exec/work/foreman/ErrorHelper.java    |   8 +-
>  .../exec/store/TestParquetPhysicalPlan.java     |  55 +++++-
>  .../store/parquet/ParquetRecordReaderTest.java  |  52 +++++-
>  .../parquet_scan_union_screen_physical.json     |   5 +-
>  9 files changed, 257 insertions(+), 148 deletions(-)
> ----------------------------------------------------------------------
>
>
>
> http://git-wip-us.apache.org/repos/asf/incubator-drill/blob/7edd3617/sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Wrapper.java
> ----------------------------------------------------------------------
> diff --git
> a/sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Wrapper.java
> b/sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Wrapper.java
> index d5a24b0..8c4b0b4 100644
> ---
> a/sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Wrapper.java
> +++
> b/sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Wrapper.java
> @@ -151,15 +151,12 @@ public class Wrapper {
>        for (int i = start; i < start + width; i++) {
>          endpoints.add(all.get(i % div));
>        }
> -    } else if (values.size() < width) {
> -      throw new NotImplementedException(
> -          "Haven't implemented a scenario where we have some node
> affinity but the affinity list is smaller than the expected width.");
>      } else {
>        // get nodes with highest affinity.
>        Collections.sort(values);
>        values = Lists.reverse(values);
>        for (int i = 0; i < width; i++) {
> -        endpoints.add(values.get(i).getEndpoint());
> +        endpoints.add(values.get(i%values.size()).getEndpoint());
>        }
>      }
>
>
>
> http://git-wip-us.apache.org/repos/asf/incubator-drill/blob/7edd3617/sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/store/AffinityCalculator.java
Regards,
Tanujit
+
Tanujit Ghosh 2013-08-24, 02:52