Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Drill, mail # dev - Out of Memory Error on tests


+
Tommy Guy 2013-11-16, 06:44
+
Timothy Chen 2013-11-16, 06:55
+
Tommy Guy 2013-11-16, 07:37
+
Timothy Chen 2013-11-16, 07:56
Copy link to this message
-
Re: Out of Memory Error on tests
Tommy Guy 2013-11-16, 23:27
Ah - I tried it on a new, 64-bit VM and the tests passed.

Tommy
On Fri, Nov 15, 2013 at 11:56 PM, Timothy Chen <[EMAIL PROTECTED]> wrote:

> That's what I thought took
>
> Of course it's possible there is a bug in your reader allocating too large
> or too many times memory.
>
> Tim
>
> Sent from my iPhone
>
> > On Nov 15, 2013, at 9:37 PM, Tommy Guy <[EMAIL PROTECTED]> wrote:
> >
> > Odd, aren't we setting this in pom.xml:
> >
> >        <plugin>
> >          <artifactId>maven-surefire-plugin</artifactId>
> >          <version>2.15</version>
> >          <configuration>
> >            <argLine>-XX:MaxDirectMemorySize=4096M </argLine>
> >          </configuration>
> >        </plugin>
> >
> > Thanks, Tim.
> >
> > Tommy
> >
> >
> >
> >> On Fri, Nov 15, 2013 at 10:55 PM, Timothy Chen <[EMAIL PROTECTED]>
> wrote:
> >>
> >> I forgot what the flag is but you also want to set your direct memory to
> >> allocate at a larger size too.
> >>
> >> Tim
> >>
> >> Sent from my iPhone
> >>
> >>> On Nov 15, 2013, at 8:44 PM, Tommy Guy <[EMAIL PROTECTED]>
> wrote:
> >>>
> >>> Hello everyone,
> >>>
> >>> I'm working on a CSV parser for Drill, and I'm running into trouble
> with
> >>> the tests. I've set -Xmx=2048m and error persists. I've tried this on
> >>> Ubuntu 11.10 and 13.10 (13.10 running in Vagrant)
> >>>
> >>> Anyone seen this before?
> >>>
> >>> Tommy
> >>>
> >>> Here's an example stack trace
> >>> from
> >>
> org.apache.drill.exec.physical.impl.TestSimpleFunctions.testByteSubstring
> >>>
> >>> java.lang.UnsupportedOperationException: null
> >>> at
> io.netty.buffer.PoolArenaL$DirectArena.newByteBuf(PoolArenaL.java:400)
> >>> at io.netty.buffer.PoolArenaL.allocate(PoolArenaL.java:94)
> >>> at
> >>
> io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:236)
> >>> at
> >>
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:132)
> >>> at
> >>
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:123)
> >>> at
> >>
> org.apache.drill.exec.memory.DirectBufferAllocator.buffer(DirectBufferAllocator.java:35)
> >>> at
> org.apache.drill.exec.vector.IntVector.allocateNew(IntVector.java:100)
> >>> at
> >>
> org.apache.drill.exec.vector.AllocationHelper.allocate(AllocationHelper.java:29)
> >>> at
> >>
> org.apache.drill.exec.store.mock.MockRecordReader.getVector(MockRecordReader.java:63)
> >>> at
> >>
> org.apache.drill.exec.store.mock.MockRecordReader.setup(MockRecordReader.java:78)
> >>> at
> >> org.apache.drill.exec.physical.impl.ScanBatch.<init>(ScanBatch.java:65)
> >>> at
> >>
> org.apache.drill.exec.store.mock.MockScanBatchCreator.getBatch(MockScanBatchCreator.java:44)
> >>> at
> >>
> org.apache.drill.exec.physical.impl.ImplCreator.visitSubScan(ImplCreator.java:105)
> >>> at
> >>
> org.apache.drill.exec.physical.impl.ImplCreator.visitSubScan(ImplCreator.java:60)
> >>> at
> >>
> org.apache.drill.exec.store.mock.MockSubScanPOP.accept(MockSubScanPOP.java:94)
> >>> at
> >>
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:195)
> >>> at
> >>
> org.apache.drill.exec.physical.impl.ImplCreator.visitIteratorValidator(ImplCreator.java:202)
> >>> at
> >>
> org.apache.drill.exec.physical.impl.ImplCreator.visitIteratorValidator(ImplCreator.java:60)
> >>> at
> >>
> org.apache.drill.exec.physical.config.IteratorValidator.accept(IteratorValidator.java:40)
> >>> at
> >>
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:195)
> >>> at
> >>
> org.apache.drill.exec.physical.impl.ImplCreator.visitProject(ImplCreator.java:92)
> >>> at
> >>
> org.apache.drill.exec.physical.impl.ImplCreator.visitProject(ImplCreator.java:60)
> >>> at
> org.apache.drill.exec.physical.config.Project.accept(Project.java:51)
> >>> at
> >>
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:195)
> >>> at
> >>
> org.apache.drill.exec.physical.impl.ImplCreator.visitIteratorValidator(ImplCreator.java:202)