Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # dev >> 0.20.205.0 Release Candidate 1 Testing


+
Scott Bush 2011-10-04, 19:40
+
Matt Foley 2011-10-04, 22:41
+
Stack 2011-10-04, 23:40
Copy link to this message
-
Re: 0.20.205.0 Release Candidate 1 Testing
Stack,

Have you turned on the append supported flag in the configuration of HDFS?
Also can you please provide steps to duplicate this problem?

Regards,
Suresh

On Tue, Oct 4, 2011 at 4:40 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Tue, Oct 4, 2011 at 3:41 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
> > I am going to spin an RC2 early tomorrow.  Does anyone have other issues
> > they consider critical for 205.0?
>
> I've been playing with it.
>
> Recovering the lease on an open file (An HBase WAL) the length is
> always zero and I don't seem to be able to recover any edits from the
> file we writing at time of the crash:
>
> 2011-10-04 21:17:04,486 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 34 of 34:
> hdfs://sv4r11s38:7000/hbase/.logs/sv4r8s38,7003,1317760866490/sv4r8s38%3A7003.1317762914728,
> length=0
> 2011-10-04 21:17:04,486 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
>
> hdfs://sv4r11s38:7000/hbase/.logs/sv4r8s38,7003,1317760866490/sv4r8s38%3A7003.1317762914728
> 2011-10-04 21:17:05,487 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
>
> hdfs://sv4r11s38:7000/hbase/.logs/sv4r8s38,7003,1317760866490/sv4r8s38%3A7003.1317762914728
> 2011-10-04 21:17:05,488 WARN
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: File
>
> hdfs://sv4r11s38:7000/hbase/.logs/sv4r8s38,7003,1317760866490/sv4r8s38%3A7003.1317762914728
> might be still open, length is 0
>
> Its probably me misconfiguring 205 compared to 0.20-append.
>
> I got some of these tooo though I'd just opened the file a few seconds
> earlier:
>
> 2011-10-04 21:16:28,439 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
>
> path=hdfs://sv4r11s38:7000/hbase/TestTable/62ff2cb514838519e5fa4282a8af4c35/recovered.edits/0000000000000008111
> region=62ff2cb514838519e5fa4282a8af4c35
>
> ....
>
> 2011-10-04 21:17:06,883 ERROR
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Couldn't close
> log at
> hdfs://sv4r11s38:7000/hbase/TestTable/62ff2cb514838519e5fa4282a8af4c35/recovered.edits/0000000000000008111
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease
> on
> /hbase/TestTable/62ff2cb514838519e5fa4282a8af4c35/recovered.edits/0000000000000008111
> File does not exist. [Lease.  Holder:
> DFSClient_hb_m_sv4r11s38:7001_1317760883384, pendingcreates: 3]
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1604)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1595)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:1650)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:1638)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:682)
>        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at javax.security.auth.Subject.doAs(Subject.java:396)
>        at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
>
>        at org.apache.hadoop.ipc.Client.call(Client.java:1066)
>        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
>        at $Proxy6.complete(Unknown Source)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
+
Stack 2011-10-05, 05:05
+
Stack 2011-10-05, 06:24
+
Stack 2011-10-06, 06:09
+
Suresh Srinivas 2011-10-06, 07:50
+
Stack 2011-10-06, 15:05
+
Matt Foley 2011-10-06, 17:18
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB