Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> MR hbase export is failing


+
Jeff Whiting 2012-07-24, 18:52
+
Jimmy Xiang 2012-07-24, 19:03
Copy link to this message
-
Re: MR hbase export is failing
I've seen this when writing when exporting to s3 and assumed it was
related to write performance. We set hbase.regionserver.lease.period to
the same values as the task timeout and it helped reduce the # of failures
though we still get occasional task timeouts. I haven't seen this when
writing to local HDFS though. Are any of your rows particularly wide? How
loaded are your data nodes when running the export? Are you running too
many mappers per node and starving hbase?

On 7/24/12 2:52 PM, "Jeff Whiting" <[EMAIL PROTECTED]> wrote:

>What would cause a scanner timeout exception?  Is hdfs too slow?  Do I
>just increase the scanner
>timeout or is there a better approach.
>
>Thanks,
>~Jeff
>
>running: hadoop jar /usr/lib/hbase/hbase-0.90.1-CDH3B4.jar export -D
>dfs.replication=2 -D
>mapred.output.compress=true -D
>mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec
>-D
>mapred.output.compression.type=BLOCK response /hbaseBackup/
>partial/2012-07-20_2012-07-21/response 1 1342742400000 1342828800000
>org.apache.hadoop.hbase.client.ScannerTimeoutException: 61294ms passed
>since the last invocation,
>timeout is currently set to 60000
>         at
>org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1114)
>         at
>org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(Table
>RecordReaderImpl.java:143)
>         at
>org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableReco
>rdReader.java:142)
>         at
>org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapT
>ask.java:455)
>         at
>org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
>         at
>org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:646)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:240)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at
>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>java:1115)
>         at org.apache.hadoop.mapred.Child.main(Child.java:234)
>Caused by: org.apache.hadoop.hbase.UnknownScannerException:
>org.apache.hadoop.hbase.UnknownScannerException: Name:
>-3564751891935236449
>         at
>org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java
>:1795)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>pl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
>org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>         at
>org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>Method)
>         at
>sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcc
>essorImpl.java:39)
>         at
>sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstr
>uctorAccessorImpl.java:27)
>         at
>java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at
>org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(Remot
>eExceptionHandler.java:96)
>         at
>org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:8
>3)
>         at
>org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:3
>8)
>         at
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n.getRegionServerWithRetries(HConnectionManager.java:1000)
>         at
>org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1100)
>         ... 12 more
>
>--
>Jeff Whiting
>Qualtrics Senior Software Engineer
>[EMAIL PROTECTED]
>
+
Ooh Rong 2012-07-25, 02:00
+
Jeff Whiting 2012-07-25, 17:34