Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill >> mail # dev >> Out of Memory Error on tests


Copy link to this message
-
Re: Out of Memory Error on tests
I forgot what the flag is but you also want to set your direct memory to allocate at a larger size too.

Tim

Sent from my iPhone

> On Nov 15, 2013, at 8:44 PM, Tommy Guy <[EMAIL PROTECTED]> wrote:
>
> Hello everyone,
>
> I'm working on a CSV parser for Drill, and I'm running into trouble with
> the tests. I've set -Xmx=2048m and error persists. I've tried this on
> Ubuntu 11.10 and 13.10 (13.10 running in Vagrant)
>
> Anyone seen this before?
>
> Tommy
>
> Here's an example stack trace
> from org.apache.drill.exec.physical.impl.TestSimpleFunctions.testByteSubstring
>
> java.lang.UnsupportedOperationException: null
> at io.netty.buffer.PoolArenaL$DirectArena.newByteBuf(PoolArenaL.java:400)
> at io.netty.buffer.PoolArenaL.allocate(PoolArenaL.java:94)
> at
> io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:236)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:132)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:123)
> at
> org.apache.drill.exec.memory.DirectBufferAllocator.buffer(DirectBufferAllocator.java:35)
> at org.apache.drill.exec.vector.IntVector.allocateNew(IntVector.java:100)
> at
> org.apache.drill.exec.vector.AllocationHelper.allocate(AllocationHelper.java:29)
> at
> org.apache.drill.exec.store.mock.MockRecordReader.getVector(MockRecordReader.java:63)
> at
> org.apache.drill.exec.store.mock.MockRecordReader.setup(MockRecordReader.java:78)
> at org.apache.drill.exec.physical.impl.ScanBatch.<init>(ScanBatch.java:65)
> at
> org.apache.drill.exec.store.mock.MockScanBatchCreator.getBatch(MockScanBatchCreator.java:44)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.visitSubScan(ImplCreator.java:105)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.visitSubScan(ImplCreator.java:60)
> at
> org.apache.drill.exec.store.mock.MockSubScanPOP.accept(MockSubScanPOP.java:94)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:195)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.visitIteratorValidator(ImplCreator.java:202)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.visitIteratorValidator(ImplCreator.java:60)
> at
> org.apache.drill.exec.physical.config.IteratorValidator.accept(IteratorValidator.java:40)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:195)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.visitProject(ImplCreator.java:92)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.visitProject(ImplCreator.java:60)
> at org.apache.drill.exec.physical.config.Project.accept(Project.java:51)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:195)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.visitIteratorValidator(ImplCreator.java:202)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.visitIteratorValidator(ImplCreator.java:60)
> at
> org.apache.drill.exec.physical.config.IteratorValidator.accept(IteratorValidator.java:40)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:195)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.visitScreen(ImplCreator.java:145)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.visitScreen(ImplCreator.java:60)
> at org.apache.drill.exec.physical.config.Screen.accept(Screen.java:102)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:212)
> at
> org.apache.drill.exec.physical.impl.TestSimpleFunctions.testByteSubstring(TestSimpleFunctions.java:200)
>
> I'm also seeing the error in Lillth:
> Exception in pipeline. Closing channel between local /127.0.1.1:31010 and
> remote /127.0.0.1:43946
>
> Lillith shows:
> java.lang.OutOfMemoryError
>
> at
> org.apache.drill.exec.rpc.RpcExceptionHandler.exceptionCaught(RpcExceptionHandler.java:33)
> at
> io.netty.channel.DefaultChannelHandlerContext.invokeExceptionCaught(DefaultChannelHandlerContext.java:272)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB