+
Peter Marron 2012-11-23, 08:47
+
Nitin Pawar 2012-11-23, 08:55
+
Peter Marron 2012-11-23, 08:58
-
Re: Creating Indexes againNitin Pawar 2012-11-23, 09:30
CDH forum might be better place to answer this. I never used CDH
On Fri, Nov 23, 2012 at 2:28 PM, Peter Marron <
[EMAIL PROTECTED]> wrote:
> Hi Nitin,****
>
> ** **
>
> Can I set these parameters through the CDH management interface?****
>
> If not then what file do they need to be set in to make sure that CDH****
>
> picks them up?****
>
> ** **
>
> Peter Marron****
>
> Trillium Software UK Limited****
>
> ** **
>
> Tel : +44 (0) 118 940 7609****
>
> Fax : +44 (0) 118 940 7699****
>
> E: [EMAIL PROTECTED] <[EMAIL PROTECTED]>**
> **
>
> ** **
>
> *From:* Nitin Pawar [mailto:[EMAIL PROTECTED]]
> *Sent:* 23 November 2012 08:55
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: Creating Indexes again****
>
> ** **
>
> try increasing ulimit on your hadoop cluster as well increase the memory
> for map and reducer both by setting them up on hive ****
>
> set mapred.job.map.memory.mb=6000;****
>
> set mapred.job.reduce.memory.mb=4000;****
>
> ** **
>
> you can change the values based on the hadoop cluster you have setup ****
>
> ** **
>
> ** **
>
> On Fri, Nov 23, 2012 at 2:17 PM, Peter Marron <
> [EMAIL PROTECTED]> wrote:****
>
> Hi,****
>
> ****
>
> I’m trying to create indexes in Hive, and I’ve switched****
>
> to using CDH-4. The creation of the index is failing and****
>
> it’s pretty obvious that the reducers are running out of****
>
> heap space. When I use the web interface for the****
>
> “Hadoop reduce task list” I can find this entry:****
>
> ****
>
> Error: Java heap space****
>
> Error: GC overhead limit exceeded****
>
> org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File
> exists****
>
> at
> org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:178)*
> ***
>
> at
> org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:303)****
>
> at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:376)****
>
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)****
>
> at java.security.AccessController.doPrivileged(Native Method)****
>
> at javax.security.auth.Subject.doAs(Subject.java:396)****
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
> ****
>
> at org.apache.hadoop.mapred.Child.main(Child.java:262)****
>
> Caused by: EEXIST: File exists****
>
> at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)****
>
> at
> org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:172)*
> ***
>
> ... 7 more****
>
> ****
>
> Error: GC overhead limit exceeded****
>
> ****
>
> *Al*****
>
> ****
>
> If Also when the****
>
> If Also when the****
>
> ****
>
> ****
>
> If Also when the****
>
> ****
>
> If this e-mail shouldn’t be here and should only be on****
>
> a cloudera mailing list, please re-direct me.****
>
> ****
>
> Thanks in advance.****
>
> ****
>
> Peter Marron****
>
> Trillium Software UK Limited****
>
> ****
>
> Tel : +44 (0) 118 940 7609****
>
> Fax : +44 (0) 118 940 7699****
>
> E: [EMAIL PROTECTED] <[EMAIL PROTECTED]>**
> **
>
> ****
>
>
>
> ****
>
> ** **
>
> --
> Nitin Pawar****
>
--
Nitin Pawar
On Fri, Nov 23, 2012 at 2:28 PM, Peter Marron <
[EMAIL PROTECTED]> wrote:
> Hi Nitin,****
>
> ** **
>
> Can I set these parameters through the CDH management interface?****
>
> If not then what file do they need to be set in to make sure that CDH****
>
> picks them up?****
>
> ** **
>
> Peter Marron****
>
> Trillium Software UK Limited****
>
> ** **
>
> Tel : +44 (0) 118 940 7609****
>
> Fax : +44 (0) 118 940 7699****
>
> E: [EMAIL PROTECTED] <[EMAIL PROTECTED]>**
> **
>
> ** **
>
> *From:* Nitin Pawar [mailto:[EMAIL PROTECTED]]
> *Sent:* 23 November 2012 08:55
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: Creating Indexes again****
>
> ** **
>
> try increasing ulimit on your hadoop cluster as well increase the memory
> for map and reducer both by setting them up on hive ****
>
> set mapred.job.map.memory.mb=6000;****
>
> set mapred.job.reduce.memory.mb=4000;****
>
> ** **
>
> you can change the values based on the hadoop cluster you have setup ****
>
> ** **
>
> ** **
>
> On Fri, Nov 23, 2012 at 2:17 PM, Peter Marron <
> [EMAIL PROTECTED]> wrote:****
>
> Hi,****
>
> ****
>
> I’m trying to create indexes in Hive, and I’ve switched****
>
> to using CDH-4. The creation of the index is failing and****
>
> it’s pretty obvious that the reducers are running out of****
>
> heap space. When I use the web interface for the****
>
> “Hadoop reduce task list” I can find this entry:****
>
> ****
>
> Error: Java heap space****
>
> Error: GC overhead limit exceeded****
>
> org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File
> exists****
>
> at
> org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:178)*
> ***
>
> at
> org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:303)****
>
> at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:376)****
>
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)****
>
> at java.security.AccessController.doPrivileged(Native Method)****
>
> at javax.security.auth.Subject.doAs(Subject.java:396)****
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
> ****
>
> at org.apache.hadoop.mapred.Child.main(Child.java:262)****
>
> Caused by: EEXIST: File exists****
>
> at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)****
>
> at
> org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:172)*
> ***
>
> ... 7 more****
>
> ****
>
> Error: GC overhead limit exceeded****
>
> ****
>
> *Al*****
>
> ****
>
> If Also when the****
>
> If Also when the****
>
> ****
>
> ****
>
> If Also when the****
>
> ****
>
> If this e-mail shouldn’t be here and should only be on****
>
> a cloudera mailing list, please re-direct me.****
>
> ****
>
> Thanks in advance.****
>
> ****
>
> Peter Marron****
>
> Trillium Software UK Limited****
>
> ****
>
> Tel : +44 (0) 118 940 7609****
>
> Fax : +44 (0) 118 940 7699****
>
> E: [EMAIL PROTECTED] <[EMAIL PROTECTED]>**
> **
>
> ****
>
>
>
> ****
>
> ** **
>
> --
> Nitin Pawar****
>
--
Nitin Pawar
