Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> The dreaded Heap Space Issue on a Transform


Copy link to this message
-
The dreaded Heap Space Issue on a Transform
I am running a transform script that parses through a bunch of binary data.
In 99% of the cases it runs, it runs fine, but on certain files I get a
failure (as seen below).  Funny thing is, I can run a job with "only" the
problem source file, and it will work fine, but when as a group of files, I
get these warnings.  I guess what I am asking here is this: Where is the
heap error? Is this occurring on the nodes themselves or, since this is
where the script is emitting records (and potentially large ones at that)
and in this case my hive server running the job may be memory light, could
the issue actually be due to heap on the hive server itself?   My setup is
1 Hive node (that is woefully underpowered, under memoried, and under disk
I/Oed) and 4 beefy hadoop nodes.  I guess, my question is the heap issue on
the sender or the receiver :)
13-01-29 08:20:24,107 INFO org.apache.hadoop.hive.ql.io.CodecPool: Got
brand-new compressor
2013-01-29 08:20:24,107 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
12 forwarding 1 rows
2013-01-29 08:20:24,410 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
3 forwarding 10 rows
2013-01-29 08:20:24,410 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
4 forwarding 10 rows
2013-01-29 08:20:24,411 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
5 forwarding 10 rows
2013-01-29 08:20:24,411 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
6 forwarding 10 rows
2013-01-29 08:20:24,411 INFO org.apache.hadoop.hive.ql.exec.FilterOperator:
8 forwarding 10 rows
2013-01-29 08:20:24,411 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
9 forwarding 10 rows
2013-01-29 08:20:24,411 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
10 forwarding 10 rows
2013-01-29 08:20:24,412 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
12 forwarding 10 rows
2013-01-29 08:20:27,170 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
3 forwarding 100 rows
2013-01-29 08:20:27,170 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
4 forwarding 100 rows
2013-01-29 08:20:27,170 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
5 forwarding 100 rows
2013-01-29 08:20:27,171 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
6 forwarding 100 rows
2013-01-29 08:20:27,171 INFO org.apache.hadoop.hive.ql.exec.FilterOperator:
8 forwarding 100 rows
2013-01-29 08:20:27,171 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
9 forwarding 100 rows
2013-01-29 08:20:27,171 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
10 forwarding 100 rows
2013-01-29 08:20:27,171 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
12 forwarding 100 rows
2013-01-29 08:21:16,247 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
3 forwarding 1000 rows
2013-01-29 08:21:16,247 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
4 forwarding 1000 rows
2013-01-29 08:21:16,247 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
5 forwarding 1000 rows
2013-01-29 08:21:16,247 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
6 forwarding 1000 rows
2013-01-29 08:21:16,248 INFO org.apache.hadoop.hive.ql.exec.FilterOperator:
8 forwarding 1000 rows
2013-01-29 08:21:16,248 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
9 forwarding 1000 rows
2013-01-29 08:21:16,248 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
10 forwarding 1000 rows
2013-01-29 08:21:16,248 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
12 forwarding 1000 rows
2013-01-29 08:25:47,532 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
3 forwarding 10000 rows
2013-01-29 08:25:47,532 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
4 forwarding 10000 rows
2013-01-29 08:25:47,532 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
5 forwarding 10000 rows
2013-01-29 08:25:47,532 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
6 forwarding 10000 rows
2013-01-29 08:25:47,532 INFO org.apache.hadoop.hive.ql.exec.FilterOperator:
8 forwarding 10000 rows
2013-01-29 08:25:47,532 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
9 forwarding 10000 rows
2013-01-29 08:25:47,532 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
10 forwarding 10000 rows
2013-01-29 08:25:47,532 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
12 forwarding 10000 rows
2013-01-29 08:27:34,276 WARN org.apache.hadoop.hive.ql.exec.ScriptOperator:
Exception in StreamThread.run(): Java heap space
Cause: null
2013-01-29 08:27:34,277 WARN org.apache.hadoop.hive.ql.exec.ScriptOperator:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3209)
 at java.lang.String.<init>(String.java:215)
at java.nio.HeapCharBuffer.toString(HeapCharBuffer.java:542)
 at java.nio.CharBuffer.toString(CharBuffer.java:1157)
at org.apache.hadoop.io.Text.decode(Text.java:350)
 at org.apache.hadoop.io.Text.decode(Text.java:327)
at org.apache.hadoop.io.Text.toString(Text.java:254)
 at java.lang.String.valueOf(String.java:2826)
at java.lang.StringBuilder.append(StringBuilder.java:115)
 at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:873)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:181)
 at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.evaluate(ExprNodeGenericFuncEvaluator.java:163)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:76)
 at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
 at
org.apache.hadoop.hive.ql.exec.ScriptOperator$OutputStreamProcessor.processLine(ScriptOperator.java:477)
at
org.apache.hadoop.hive.ql.exec.ScriptOperator$StreamThread.run(ScriptOperator.java:563)

2013-01-29 08:27:34,306 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
ErrorStreamProcessor calling reporter.progress()
2013-01-29 08:27:34,307 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
StreamThread ErrorProcessor done
2013-01-29 08:27:34,307 ERROR
org.apache.hadoop.hive.ql.exec.ScriptOperator: Script failed with code 1
+
John Omernik 2013-01-30, 14:28
+
Philip Tromans 2013-01-30, 15:11
+
John Omernik 2013-01-30, 15:54
+
Richard Nadeau 2013-01-30, 16:14
+
Dean Wampler 2013-01-30, 16:40
+
John Omernik 2013-01-30, 17:35
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB