Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Never ending "Doing distributed log split" task.,


+
Jean-Marc Spaggiari 2013-08-29, 16:03
+
Ted Yu 2013-08-29, 16:43
+
Jean-Marc Spaggiari 2013-08-29, 16:50
+
Ted Yu 2013-08-29, 16:56
Copy link to this message
-
Re: Never ending "Doing distributed log split" task.,
I have not yet found an easy way to upgrade to hadoop 1.2.1 so I'm waiting
for my 2nd cluster to be ready to install it with 1.2.1 and distcp to it.
But that's outside of the scope of this discussion ;)
2013/8/29 Ted Yu <[EMAIL PROTECTED]>

> So you have HBASE-8670 in your deployment.
>
> Suggest upgrading hadoop to newer release, e.g. 1.2.1 so that the new HDFS
> improvements can be utilized.
>
> Cheers
>
>
> On Thu, Aug 29, 2013 at 9:50 AM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]> wrote:
>
> > Hadoop 1.0.4 with HBase 0.94.12-SNAPSHOT
> >
> > The file name changed since I have restarted HBase but here is what I
> have:
> > hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -ls
> > hdfs://node3:9000/hbase/.logs/node1,60020,1377793020654/
> > Found 1 items
> > -rw-r--r--   3 hbase supergroup          0 2013-08-29 12:17
> >
> >
> /hbase/.logs/node1,60020,1377793020654/node1%2C60020%2C1377793020654.1377793021892
> >
> > And I'm able to access it:
> > hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -get
> >
> >
> /hbase/.logs/node1,60020,1377793020654/node1%2C60020%2C1377793020654.1377793021892
> > .
> > hadoop@node3:~/hadoop-1.0.3$
> >
> > Oh. I just checked the UI again, and it's done. Wow! Took almost 1h. HBCK
> > report 0 inconsistencies detected. Status: OK
> >
> > So seems that I'm all fine.
> >
> > I don't know why it was so long. I will try to take a look at my
> Ganglia's
> > metrics to see if I can figure anything...
> >
> > JM
> >
> >
> >
> > 2013/8/29 Ted Yu <[EMAIL PROTECTED]>
> >
> > > What is your HBase / Hadoop version ?
> > >
> > > Can you check namenode log looking for lines related to
> > > hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-
> > > splitting/node1%2C60020%2C1377789460683.1377789462024 ?
> > >
> > > Thanks
> > >
> > >
> > > On Thu, Aug 29, 2013 at 9:03 AM, Jean-Marc Spaggiari <
> > > [EMAIL PROTECTED]> wrote:
> > >
> > > > I have restart my cluster and I'm now waiting for this task to end:
> > > >
> > > > Doing distributed log split in
> > > > [hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-splitting]
> > > >
> > > > It's running fir now 30 minutes. There was nothing running on the
> > > cluster.
> > > > No reads, no writes, nothing, for days...
> > > >
> > > > I got that on the logs:
> > > >
> > > > 2013-08-29 11:36:10,862 WARN
> > > > org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting of
> > > >
> > > >
> > >
> >
> hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-splitting/node1%2C60020%2C1377789460683.1377789462024
> > > > interrupted, resigning
> > > > java.io.InterruptedIOException
> > > >     at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:136)
> > > >     at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:54)
> > > >     at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:780)
> > > >     at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:414)
> > > >     at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:381)
> > > >     at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:112)
> > > >     at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:280)
> > > >     at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:211)
> > > >     at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:179)
> > > >     at java.lang.Thread.run(Thread.java:722)
> > > > Caused by: java.lang.InterruptedException: sleep interrupted
> > > >     at java.lang.Thread.sleep(Native Method)
> > > >     at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:118)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB