Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> 'Cannot roll edit log'


Copy link to this message
-
Re: 'Cannot roll edit log'
Further info: From the log of secondary NN:

INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Posted
URL
AAA.xxx.yyy.zzz:50070putimage=1&port=50090&machine=0.0.0.0&token=-32:1245372967:0:1343222881000:1343222711330

ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
Exception in doCheckpoint:

ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
java.io.FileNotFoundException:
http://AAA.xxx.yyy.zzz:50070/getimage?putimage=1&port=50090&machine=0.0.0.0&token=-32:1245372967:0:1343222881000:1343222711330

Where Primary NN is AAA.xxx.yyy.zzz.

The 0.0.0.0 embedded in the URL looks suspicious, but I have no idea
what file it is telling me it is missing.
On 08/15/2012 12:39 PM, Terry Healy wrote:
> This situation continues and reports every 10 minutes. I even tried
> moving the secondary NN function to a different node. I have run TCPDump
> but cannot isolate the "Connection Refused" issue. Am I correct in
> assuming that the NN will try to connect to SNN on port 50090?
>
> Any way out? Current partial dump below.
>
> 2012-08-15 12:36:09,422 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> xxx.yyy.254.254
>
> 2012-08-15 12:36:09,423 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit
> log, edits.new files already exists in all healthy directories:
>   /home/thealy/hdfs/name/current/edits.new
>
> 2012-08-15 12:36:09,879 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:thealy cause:java.net.ConnectException:
> Connection refused
>
> 2012-08-15 12:36:09,879 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:thealy cause:java.net.ConnectException:
> Connection refused
>
> 2012-08-15 12:36:09,881 WARN org.mortbay.log: /getimage:
> java.io.IOException: GetImage failed. java.net.ConnectException:
> Connection refused
>
>
>
> On 07/09/2012 01:23 PM, Terry Healy wrote:
>> Any suggestions on how to clear the error? stp-all / start-all had no
>> effect.
>>
>> On 07/03/2012 04:22 PM, Brandon Li wrote:
>>> With 1.0.2, only one checkpoint process is executed at a time. When the
>>> namenode gets an overlapping checkpointing request, it checks edit.new
>>> in its storage directories. If all of them have this file, namenode
>>> concludes the previous checkpoint process is not done yet and prints the
>>> warning message you've seen.
>>>
>>> Brandon
>>>
>>> On Tue, Jul 3, 2012 at 10:56 AM, Terry Healy <[EMAIL PROTECTED]
>>> <mailto:[EMAIL PROTECTED]>> wrote:
>>>
>>>     Running Apache 1.0.2.
>>>
>>>     The NN log is reporting that it cannot "roll the edit log" from the
>>>     secondary NN. The SecondaryNameNode is running on the system referred to
>>>     as xxx.yyy.254.238 in the log snippet below.
>>>
>>>     From the NN, I can connect to the Secondary via ssh as the user. Any
>>>     suggestions what have I got wrong here?
>>>
>>>     thanks,
>>>
>>>     Terry
>>>
>>>     INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>>     from xxx.yyy.254.238
>>>     WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll
>>>     edit log, edits.new files already exists in all healthy directories:
>>>       /home/[user]/hdfs/name/current/edits.new
>>>
>>>     ERROR org.apache.hadoop.security.UserGroupInformation:
>>>     PriviledgedActionException as:[user] cause:java.net.ConnectException:
>>>     Connection refused
>>>     ERROR org.apache.hadoop.security.UserGroupInformation:
>>>     PriviledgedActionException as:[user] cause:java.net.ConnectException:
>>>     Connection refused
>>>
>>>     WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed.
>>>     java.net.ConnectException: Connection refused
>>>             at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>             at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
>>>     ....
>>>
>>>
>>>     --
>>>     Terry Healy / [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>

Terry Healy / [EMAIL PROTECTED]
Cyber Security Operations
Brookhaven National Laboratory
Building 515, Upton N.Y. 11973
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB