Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill >> mail # dev >> Out of Memory Error on tests


Copy link to this message
-
Out of Memory Error on tests
Hello everyone,

I'm working on a CSV parser for Drill, and I'm running into trouble with
the tests. I've set -Xmx=2048m and error persists. I've tried this on
Ubuntu 11.10 and 13.10 (13.10 running in Vagrant)

Anyone seen this before?

Tommy

Here's an example stack trace
from org.apache.drill.exec.physical.impl.TestSimpleFunctions.testByteSubstring

java.lang.UnsupportedOperationException: null
at io.netty.buffer.PoolArenaL$DirectArena.newByteBuf(PoolArenaL.java:400)
at io.netty.buffer.PoolArenaL.allocate(PoolArenaL.java:94)
at
io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:236)
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:132)
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:123)
at
org.apache.drill.exec.memory.DirectBufferAllocator.buffer(DirectBufferAllocator.java:35)
at org.apache.drill.exec.vector.IntVector.allocateNew(IntVector.java:100)
at
org.apache.drill.exec.vector.AllocationHelper.allocate(AllocationHelper.java:29)
at
org.apache.drill.exec.store.mock.MockRecordReader.getVector(MockRecordReader.java:63)
at
org.apache.drill.exec.store.mock.MockRecordReader.setup(MockRecordReader.java:78)
at org.apache.drill.exec.physical.impl.ScanBatch.<init>(ScanBatch.java:65)
at
org.apache.drill.exec.store.mock.MockScanBatchCreator.getBatch(MockScanBatchCreator.java:44)
at
org.apache.drill.exec.physical.impl.ImplCreator.visitSubScan(ImplCreator.java:105)
at
org.apache.drill.exec.physical.impl.ImplCreator.visitSubScan(ImplCreator.java:60)
at
org.apache.drill.exec.store.mock.MockSubScanPOP.accept(MockSubScanPOP.java:94)
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:195)
at
org.apache.drill.exec.physical.impl.ImplCreator.visitIteratorValidator(ImplCreator.java:202)
at
org.apache.drill.exec.physical.impl.ImplCreator.visitIteratorValidator(ImplCreator.java:60)
at
org.apache.drill.exec.physical.config.IteratorValidator.accept(IteratorValidator.java:40)
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:195)
at
org.apache.drill.exec.physical.impl.ImplCreator.visitProject(ImplCreator.java:92)
at
org.apache.drill.exec.physical.impl.ImplCreator.visitProject(ImplCreator.java:60)
at org.apache.drill.exec.physical.config.Project.accept(Project.java:51)
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:195)
at
org.apache.drill.exec.physical.impl.ImplCreator.visitIteratorValidator(ImplCreator.java:202)
at
org.apache.drill.exec.physical.impl.ImplCreator.visitIteratorValidator(ImplCreator.java:60)
at
org.apache.drill.exec.physical.config.IteratorValidator.accept(IteratorValidator.java:40)
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:195)
at
org.apache.drill.exec.physical.impl.ImplCreator.visitScreen(ImplCreator.java:145)
at
org.apache.drill.exec.physical.impl.ImplCreator.visitScreen(ImplCreator.java:60)
at org.apache.drill.exec.physical.config.Screen.accept(Screen.java:102)
at
org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:212)
at
org.apache.drill.exec.physical.impl.TestSimpleFunctions.testByteSubstring(TestSimpleFunctions.java:200)

I'm also seeing the error in Lillth:
Exception in pipeline. Closing channel between local /127.0.1.1:31010 and
remote /127.0.0.1:43946

Lillith shows:
java.lang.OutOfMemoryError

at
org.apache.drill.exec.rpc.RpcExceptionHandler.exceptionCaught(RpcExceptionHandler.java:33)
at
io.netty.channel.DefaultChannelHandlerContext.invokeExceptionCaught(DefaultChannelHandlerContext.java:272)
at
io.netty.channel.DefaultChannelHandlerContext.fireExceptionCaught(DefaultChannelHandlerContext.java:250)
at
io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:60)
at
io.netty.channel.DefaultChannelHandlerContext.invokeExceptionCaught(DefaultChannelHandlerContext.java:272)
at
io.netty.channel.DefaultChannelHandlerContext.fireExceptionCaught(DefaultChannelHandlerContext.java:250)
at
io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:60)
at
io.netty.channel.DefaultChannelHandlerContext.invokeExceptionCaught(DefaultChannelHandlerContext.java:272)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB