Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Creating Indexes

Peter Marron 2012-10-31, 17:42
Shreepadma Venugopalan 2012-10-31, 18:37
Peter Marron 2012-10-31, 19:52
Shreepadma Venugopalan 2012-10-31, 21:58
Peter Marron 2012-11-01, 09:31
Dean Wampler 2012-11-01, 13:01
Bejoy KS 2012-11-01, 13:09
Dean Wampler 2012-11-01, 13:20
Peter Marron 2012-11-02, 08:44
Dean Wampler 2012-11-02, 14:02
Peter Marron 2012-11-02, 23:59
Shreepadma Venugopalan 2012-11-03, 00:06
Peter Marron 2012-11-07, 13:33
Dean Wampler 2012-11-03, 00:02
Dean Wampler 2012-11-03, 00:03
Copy link to this message
Creating indexes
(Sorry previous incomplete message sent in error.)


I'm trying to create indexes in Hive, and I've switched
to using CDH-4. The creation of the index is failing and
it's pretty obvious that the reducers are running out of
heap space. When I use the web interface for the
"Hadoop reduce task list" I can find this entry:

Error: Java heap space
Error: GC overhead limit exceeded
org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File exists
        at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:178)
        at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:303)
        at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:376)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
        at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: EEXIST: File exists
        at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
        at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:172)
        ... 7 more

Error: GC overhead limit exceeded

Also when the job is running I can look at the administration page and
I can see a line that says something like:

Cluster Summary (Heap Size is 52.12 MB/380.5 MB)

And it seems likely that 380Mb is not enough to run this job.
The thing is that I have tried to set the heap to a higher value.
I've searched in the configuration interface for "heap" and
set the values higher. This is what I see:

What am I doing wrong?
If this e-mail shouldn't be here and should only be on
a cloudera mailing list, please re-direct me.

Thanks in advance.

Peter Marron
Trillium Software UK Limited

Tel : +44 (0) 118 940 7609
Fax : +44 (0) 118 940 7699