Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Hive Query


hello Bejoy,

I have checked the logs of failour nodes on TT web Interface,

it is.

2012-07-24 15:38:45,415 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=SHUFFLE, sessionId2012-07-24 15:38:45,554 INFO org.apache.hadoop.mapred.ReduceTask: ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
2012-07-24 15:38:45,562 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 Thread started: Thread for merging in memory files
2012-07-24 15:38:45,562 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 Thread started: Thread for merging on-disk files
2012-07-24 15:38:45,563 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 Thread waiting: Thread for merging on-disk files
2012-07-24 15:38:45,564 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 Need another 1 map output(s) where 0 is already in progress
2012-07-24 15:38:45,564 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 Thread started: Thread for polling Map Completion Events
2012-07-24 15:38:45,565 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 Scheduled 0 outputs (0 slow hosts and0 dup hosts)
2012-07-24 15:38:45,569 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3: Got 1 new map-outputs
2012-07-24 15:38:50,565 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 Scheduled 1 outputs (0 slow hosts and0 dup hosts)
2012-07-24 15:38:50,632 WARN org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 copy failed: attempt_201207241536_0002_m_000000_0 from 10.203.33.81
2012-07-24 15:38:50,634 WARN org.apache.hadoop.mapred.ReduceTask: java.io.IOException: Server returned HTTP response code: 407 for URL: http://10.203.33.81:50060/mapOutput?job=job_201207241536_0002&map=attempt_201207241536_0002_m_000000_0&reduce=0
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1491)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1485)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1139)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1447)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1349)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1261)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1195)
Caused by: java.io.IOException: Server returned HTTP response code: 407 for URL: http://10.203.33.81:50060/mapOutput?job=job_201207241536_0002&map=attempt_201207241536_0002_m_000000_0&reduce=0
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
    ... 4 more

2012-07-24 15:38:50,635 INFO org.apache.hadoop.mapred.ReduceTask: Task attempt_201207241536_0002_r_000000_3: Failed fetch #1 from attempt_201207241536_0002_m_000000_0
2012-07-24 15:38:50,635 WARN org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 adding host 10.203.33.81 to penalty box, next contact in 4 seconds
2012-07-24 15:38:50,635 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3: Got 1 map-outputs from previous failures
2012-07-24 15:38:55,635 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 Scheduled 1 outputs (0 slow hosts and0 dup hosts)
2012-07-24 15:38:55,689 WARN org.apache.hadoop.mapred.ReduceTask: attempt_201207241536_0002_r_000000_3 copy failed: attempt_201207241536_0002_m_000000_0 from 10.203.33.81
2012-07-24 15:38:55,689 WARN org.apache.hadoop.mapred.ReduceTask: java.io.IOException: Server returned HTTP response code: 407 for URL: http://10.203.33.81:50060/mapOutput?job=job_201207241536_0002&map=attempt_201207241536_0002_m_000000_0&reduce=0
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1491)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1485)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1139)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1447)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1349)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1261)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1195)
Caused by: java.io.IOException: Server returned HTTP response code: 407 for URL: http://10.203.33.81:50060/mapOutput?job=job_201207241536_0002&map=attempt_201207241536_0002_m_000000_0&reduce=0
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
    ... 4 more

2012-07-24 15:38:55,690 INFO org.apache.hadoop.mapred.ReduceTask: Task attempt_201207241536_0002_r_000000_3