Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Setting input paths


Copy link to this message
-
Re: Setting input paths
hadoopman 2011-04-06, 16:23
I have a process which  is loading data into hive hourly.  Loading data
hourly isn't a problem however when I load historical data say 24-48
hours I receive the below error msg.  In googling I've come across some
suggestions that jvm memory needs to be increased.  Are there any other
options or is that pretty much it?

I appreciate any help with this one.  To get around the problem I've
loaded fewer historical logs which works great but isn't what I had in
mind :-)

Thanks a bunch!

---

2011-04-05 15:33:38,400 INFO
org.apache.hadoop.hive.ql.exec.SelectOperator: 7 finished. closing...
2011-04-05 15:33:38,400 INFO
org.apache.hadoop.hive.ql.exec.SelectOperator: 7 forwarded 0 rows
2011-04-05 15:33:38,400 INFO
org.apache.hadoop.hive.ql.exec.GroupByOperator: 8 finished. closing...
2011-04-05 15:33:38,400 INFO
org.apache.hadoop.hive.ql.exec.GroupByOperator: 8 forwarded 0 rows
2011-04-05 15:33:38,401 WARN
org.apache.hadoop.hive.ql.exec.GroupByOperator: Begin Hash Table flush
at close: size = 0
2011-04-05 15:33:38,401 INFO
org.apache.hadoop.hive.ql.exec.FileSinkOperator: 9 finished. closing...
2011-04-05 15:33:38,401 INFO
org.apache.hadoop.hive.ql.exec.FileSinkOperator: 9 forwarded 0 rows
2011-04-05 15:33:38,401 INFO
org.apache.hadoop.hive.ql.exec.FileSinkOperator: Final Path: FS
hdfs://namenoden1:9000
/tmp/hive-etl/hive_2011-04-05_15-25-32_126_7118636463039801851/_tmp.-mr-10004/000049_0
2011-04-05 15:33:38,401 INFO
org.apache.hadoop.hive.ql.exec.FileSinkOperator: Writing to temp file: FS
hdfs://namenoden1:9000/tmp/hive-etl/hive_2011-04-05_15-25-32_126_7118636463039801851/_tmp.-mr-10004/_tmp.000049_0
2011-04-05 15:33:38,401 INFO
org.apache.hadoop.hive.ql.exec.FileSinkOperator: New Final Path: FS
hdfs://namenoden1:9000
/tmp/hive-etl/hive_2011-04-05_15-25-32_126_7118636463039801851/_tmp.-mr-10004/000049_0
2011-04-05 15:33:38,448 INFO org.apache.hadoop.io.compress.CodecPool:
Got brand-new compressor
2011-04-05 15:33:38,495 INFO
org.apache.hadoop.hive.ql.exec.GroupByOperator: 8 Close done
2011-04-05 15:33:38,495 INFO
org.apache.hadoop.hive.ql.exec.SelectOperator: 7 Close done
2011-04-05 15:33:38,495 INFO
org.apache.hadoop.hive.ql.exec.FilterOperator: 6 Close done
2011-04-05 15:33:38,495 INFO
org.apache.hadoop.hive.ql.exec.SelectOperator: 1 Close done
2011-04-05 15:33:38,495 INFO
org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 Close done
2011-04-05 15:33:38,495 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
13 Close done
2011-04-05 15:33:38,495 INFO ExecMapper: ExecMapper: processed 188614
rows: used memory = 720968432
2011-04-05 15:33:38,501 FATAL org.apache.hadoop.mapred.Child: Error
running child : java.lang.OutOfMemoryError: Java
heap space
                 at org.apache.hadoop.io.Text.setCapacity(Text.java:240)
                 at org.apache.hadoop.io.Text.append(Text.java:216)
                 at
org.apache.hadoop.util.LineReader.readLine(LineReader.java:159)
                 at
org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
                 at
org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
                 at
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:66)
                 at
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:32)
                 at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
                 at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:202)
                 at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:186)
                 at
org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
                 at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:383)
                 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:317)
                 at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
                 at java.security.AccessController.doPrivileged(Native
Method)
                 at javax.security.auth.Subject.doAs(Subject.java:396)
                 at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
                 at org.apache.hadoop.mapred.Child.main(Child.java:211)