Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Hive Query


Thank you Bejoy, Thank you very much:-).

Yes my n/w setup was with proxy authentication, if I do remove proxy for HTTP, HTTPS then it works but the Internet goes down.

I set the proxy bypass for machine and it working. :-)

hip hip hurry ;-)

Thanks a Lot :-)
Yogesh Kumar
________________________________
From: Bejoy Ks [[EMAIL PROTECTED]]
Sent: Tuesday, July 24, 2012 4:18 PM
To: [EMAIL PROTECTED]
Subject: Re: Hive Query

Hi Yogesh

Did you try out the suggested changes?

1. Increase the value of tasktracker.http.threads  (this to be done at TT level and not on job level, restart TT)
2. Increase the value mapred.reduce.parallel.copies

You need to add the new values for these properties in mapred-site.xml and restart all the TTs .  (I'm not very sure this will resolve your issue, Just a suggestion).

Also I as Nitin suggested please work on proxy authentication as well.
Regards
Bejoy KS

________________________________
From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Sent: Tuesday, July 24, 2012 3:54 PM
Subject: RE: Hive Query

hello Bejoy,

I have checked the logs of failour nodes on TT web Interface,

it is.

2012-07-24 15:38:45,415 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=SHUFFLE, sessionId2012-07-24 15:38:45,554 INFO org.apache.hadoop.mapred.ReduceTask: ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
2012-07-24 15:38:45,562 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 Thread started: Thread for merging in memory files
2012-07-24 15:38:45,562 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 Thread started: Thread for merging on-disk files
2012-07-24 15:38:45,563 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 Thread waiting: Thread for merging on-disk files
2012-07-24 15:38:45,564 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 Need another 1 map output(s) where 0 is already in progress
2012-07-24 15:38:45,564 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 Thread started: Thread for polling Map Completion Events
2012-07-24 15:38:45,565 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 Scheduled 0 outputs (0 slow hosts and0 dup hosts)
2012-07-24 15:38:45,569 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3: Got 1 new map-outputs
2012-07-24 15:38:50,565 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 Scheduled 1 outputs (0 slow hosts and0 dup hosts)
2012-07-24 15:38:50,632 WARN org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 copy failed: attempt_201207241536_0002_m_000000_0 from 10.203.33.81
2012-07-24 15:38:50,634 WARN org.apache.hadoop.mapred.ReduceTask: java.io.IOException: Server returned HTTP response code: 407 for URL: http://10.203.33.81:50060/mapOutput?job=job_201207241536_0002&map=attempt_201207241536_0002_m_000000_0&reduce=0
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1491)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1485)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1139)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1447)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1349)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1261)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1195)
Caused by: java.io.IOException: Server returned HTTP response code: 407 for URL: http://10.203.33.81:50060/mapOutput?job=job_201207241536_0002&map=attempt_201207241536_0002_m_000000_0&reduce=0
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
    ... 4 more

2012-07-24 15:38:50,635 INFO org.apache.hadoop.mapred.ReduceTask: Task attempt_201207241536_0002_r_000000_3: Failed fetch #1 from attempt_201207241536_0002_m_000000_0
2012-07-24 15:38:50,635 WARN org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 adding host 10.203.33.81 to penalty box, next contact in 4 seconds
2012-07-24 15:38:50,635 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3: Got 1 map-outputs from previous failures
2012-07-24 15:38:55,635 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 Scheduled 1 outputs (0 slow hosts and0 dup hosts)
2012-07-24 15:38:55,689 WARN org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 copy failed: attempt_201207241536_0002_m_000000_0 from 10.203.33.81
2012-07-24 15:38:55,689 WARN org.apache.hadoop.mapred.ReduceTask: java.io.IOException: Server returned HTTP response code: 407 for URL: http://10.203.33.81:50060/mapOutput?job=job_201207241536_0002&map=attempt_201207241536_0002_m_000000_0&reduce=0
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConne