Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Child Error


+
Jim Twensky 2013-05-24, 00:14
Hi Jim,

Which JVM are you using?

I don't think you have any memory issue. Else you will have got some OOME...

JM

2013/5/24 Jim Twensky <[EMAIL PROTECTED]>

> Hi again, in addition to my previous post, I was able to get some error
> logs from the task tracker/data node this morning and looks like it might
> be a jetty issue:
>
> 2013-05-23 19:59:20,595 WARN org.apache.hadoop.mapred.TaskLog: Failed to
> retrieve stdout log for task: attempt_201305231647_0007_m_001096_0
> java.io.IOException: Owner 'jim' for path
> /var/tmp/jim/hadoop-logs/userlogs/job_201305231647_0007/attempt_201305231647_0007_m_001096_0/stdout
> did not match expected owner '10929'
>   at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177)
>   at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117)
>   at org.apache.hadoop.mapred.TaskLog$Reader.<init>(TaskLog.java:455)
>   at
> org.apache.hadoop.mapred.TaskLogServlet.printTaskLog(TaskLogServlet.java:81)
>   at org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>   at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>   at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>   at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:848)
>   at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>   at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>
>
> I am wondering if I am hitting MAPREDUCE-2389<https://issues.apache.org/jira/browse/MAPREDUCE-2389>If so, how do I downgrade my jetty version? Should I just replace the jetty
> jar file in the lib directory with an earlier version and restart my
> cluster?
>
> Thank you.
>
>
>
>
> On Thu, May 23, 2013 at 7:14 PM, Jim Twensky <[EMAIL PROTECTED]>wrote:
>
>> Hello, I have a 20 node Hadoop cluster where each node has 8GB memory and
>> an 8-core processor. I sometimes get the following error on a random basis:
>>
>>
>>
>> -----------------------------------------------------------------------------------------------------------
>>
>> Exception in thread "main" java.io.IOException: Exception reading file:/var/tmp/jim/hadoop-jim/mapred/local/taskTracker/jim/jobcache/job_201305231647_0005/jobToken
>> at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
>> at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:165)
>> at org.apache.hadoop.mapred.Child.main(Child.java:92)
>> Caused by: java.io.IOException: failure to login
>> at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:501)
>> at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:463)
>> at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1519)
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1420)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
>> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
>> at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
>> ... 2 more
>> Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
>> at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:70)
>> at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:132)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> ......
>>
>>
>> -----------------------------------------------------------------------------------------------------------
>>
>> This does not always happen but I see a pattern when the intermediate
+
Jean-Marc Spaggiari 2013-05-28, 14:24
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB