Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> hadoop-1.0.0 and errors with log.index


Copy link to this message
-
Re: hadoop-1.0.0 and errors with log.index
Anything in TaskTracker logs ?

On Jan 31, 2012, at 10:18 AM, Markus Jelsma wrote:

> In our case, which seems to be the same problem, the web UI does not show
> anything useful except the first line of the stack trace:
>
> 2012-01-03 21:16:27,256 WARN org.apache.hadoop.mapred.TaskLog: Failed to
> retrieve stdout log for task: attempt_201201031651_0008_m_000233_0
>
> Only the task tracker log shows a full stack trace. This happened on 1.0.0 and
> 0.20.205.0 but not 0.20.203.0.
>
> 2012-01-03 21:16:27,256 WARN org.apache.hadoop.mapred.TaskLog: Failed to
> retrieve stdout log for task: attempt_201201031651_0008_m_000233_0
> java.io.FileNotFoundException:
> /opt/hadoop/hadoop-0.20.205.0/libexec/../logs/userlogs/job_201201031651_0008/attempt_201201031651_0008_m_000233_0/log.index
> (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.(SecureIOUtils.java:102)
> at
> org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:187)
> at org.apache.hadoop.mapred.TaskLog$Reader.(TaskLogServlet.java:81)
> at
> org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>
>
>> Actually, all that is telling you is that the task failed and the
>> job-client couldn't display the logs.
>>
>> Can you check the JT web-ui and see why the task failed ?
>>
>> If you don't see anything there, you can try see the TaskTracker logs on
>> the node on which the task ran.
>>
>> Arun
>>
>> On Jan 31, 2012, at 3:21 AM, Marcin Cylke wrote:
>>> Hi
>>>
>>> I've upgraded my hadoop cluster to version 1.0.0. The upgrade process
>>> went relatively smoothly but it rendered the cluster inoperable due to
>>> errors in jobtrackers operation:
>>>
>>> # in job output
>>> Error reading task
>>> outputhttp://hadoop4:50060/tasklog?plaintext=true&attemptid=attempt_20120
>>> 1311241_0003_m_000004_2&filter=stdout
>>>
>>> # in each of the jobtrackers' logs
>>> WARN org.apache.hadoop.mapred.TaskLog: Failed to retrieve stderr log for
>>> task: attempt_201201311241_0003_r_000000_1
>>> java.io.FileNotFoundException:
>>> /usr/lib/hadoop-1.0.0/libexec/../logs/userlogs/job_201201311241_0003/atte
>>> mpt_201201311241_0003_r_000000_1/log.index (No such file or directory)
>>>
>>>        at java.io.FileInputStream.open(Native Method)
>>>
>>> These errors seem related to this two problems:
>>>
>>> http://grokbase.com/t/hadoop.apache.org/mapreduce-user/2012/01/error-read
>>> ing-task-output-and-log-filenotfoundexceptions/03mjwctewcnxlgp2jkcrhvsgep
>>> 4e
>>>
>>> https://issues.apache.org/jira/browse/MAPREDUCE-2846
>>>
>>> But I've looked into the source code and the fix from MAPREDUCE-2846 is
>>> there. Perhaps there is some other reason?

Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB