Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Creating Indexes


+
Peter Marron 2012-10-31, 17:42
+
Shreepadma Venugopalan 2012-10-31, 18:37
+
Peter Marron 2012-10-31, 19:52
+
Shreepadma Venugopalan 2012-10-31, 21:58
+
Peter Marron 2012-11-01, 09:31
+
Dean Wampler 2012-11-01, 13:01
+
Bejoy KS 2012-11-01, 13:09
+
Dean Wampler 2012-11-01, 13:20
+
Peter Marron 2012-11-02, 08:44
+
Dean Wampler 2012-11-02, 14:02
+
Peter Marron 2012-11-02, 23:59
+
Shreepadma Venugopalan 2012-11-03, 00:06
+
Peter Marron 2012-11-07, 13:33
+
Dean Wampler 2012-11-03, 00:02
+
Dean Wampler 2012-11-03, 00:03
Copy link to this message
-
Creating indexes
(Sorry previous incomplete message sent in error.)

Hi,

I'm trying to create indexes in Hive, and I've switched
to using CDH-4. The creation of the index is failing and
it's pretty obvious that the reducers are running out of
heap space. When I use the web interface for the
"Hadoop reduce task list" I can find this entry:

Error: Java heap space
Error: GC overhead limit exceeded
org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File exists
        at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:178)
        at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:303)
        at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:376)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
        at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: EEXIST: File exists
        at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
        at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:172)
        ... 7 more

Error: GC overhead limit exceeded

Also when the job is running I can look at the administration page and
I can see a line that says something like:

Cluster Summary (Heap Size is 52.12 MB/380.5 MB)

And it seems likely that 380Mb is not enough to run this job.
The thing is that I have tried to set the heap to a higher value.
I've searched in the configuration interface for "heap" and
set the values higher. This is what I see:

[cid:[EMAIL PROTECTED]AD20]
What am I doing wrong?
If this e-mail shouldn't be here and should only be on
a cloudera mailing list, please re-direct me.

Thanks in advance.

Peter Marron
Trillium Software UK Limited

Tel : +44 (0) 118 940 7609
Fax : +44 (0) 118 940 7699
E: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB