-
Re: Regarding ScannerTimeout ExceptionMarcos Ortiz 2013-02-16, 17:33
Regards, Vidosh. My responses in line
On 02/16/2013 01:03 AM, Vidosh Sahu wrote:
> Hi,
>
> I am using HBase 0.92.1-cdh4.1.1. There is mapper job which do a lot of
> operation. This job is getting failed with the below operation. It looks
> like the scanner operation is timed out. How to increase it using Cloudera
> Manager Web Console. Can anyone please help me out.
Well, this questions should be in CDH mailing list ([EMAIL PROTECTED]).
Another thing is that Scanner�s performance have seen a lot of improvements
until the stable release (which is 0.94.5 right now, thanks to Lars H.
for the information).
I remembered a great talk by Mikhail Bautin called "Optimizing HBase
scanner performance"
in the last HBaseCon who explained all things done in this particular
topic, essentially:[1]
- HBASE-4433: Avoid extra next if done with row/column
- HBASE-4434: Don't do HFile Scanner next() unless the next KV is needed
- HBASE-2794: Multi-column Bloom Filters
- HBASE-4465: Lazy Seek
- HBASE-4469: Utilize existing ROWCOL Bloom filter
- HBASE-4532: Added a separate ROW-only Bloom filter for DeleteFamily
- HBASE-4585: Seek on deleted KV
- HBASE-4962: Top-of-the-column seek
So, you can see that you should upgrade your HBase installation. CDH 4.2
is out, and CDH 5.0 will be available very soon.[2]
[1] http://www.slideshare.net/cloudera/tag/hbasecon-2012
[2]
http://blog.cloudera.com/blog/2012/11/apache-hbase-assignmentmanager-improvements/
>
> 2013-02-16 05:30:45,978 [Thread-158] WARN
> org.apache.hadoop.mapred.LocalJobRunner - job_local_0005
> org.apache.hadoop.hbase.client.ScannerTimeoutException: 614527ms passed
> since the last invocation, timeout is currently set to 60000
> at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1302)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:133)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:142)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423)
> at
> org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> Caused by: org.apache.hadoop.hbase.UnknownScannerException:
> org.apache.hadoop.hbase.UnknownScannerException: Name: -5689607602334883791
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2110)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:84)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
Marcos Ortiz Valmaseda,
Product Manager && Data Scientist at UCI
Blog: http://marcosluis2186.posterous.com
Twitter: @marcosluis2186 <http://twitter.com/marcosluis2186>
On 02/16/2013 01:03 AM, Vidosh Sahu wrote:
> Hi,
>
> I am using HBase 0.92.1-cdh4.1.1. There is mapper job which do a lot of
> operation. This job is getting failed with the below operation. It looks
> like the scanner operation is timed out. How to increase it using Cloudera
> Manager Web Console. Can anyone please help me out.
Well, this questions should be in CDH mailing list ([EMAIL PROTECTED]).
Another thing is that Scanner�s performance have seen a lot of improvements
until the stable release (which is 0.94.5 right now, thanks to Lars H.
for the information).
I remembered a great talk by Mikhail Bautin called "Optimizing HBase
scanner performance"
in the last HBaseCon who explained all things done in this particular
topic, essentially:[1]
- HBASE-4433: Avoid extra next if done with row/column
- HBASE-4434: Don't do HFile Scanner next() unless the next KV is needed
- HBASE-2794: Multi-column Bloom Filters
- HBASE-4465: Lazy Seek
- HBASE-4469: Utilize existing ROWCOL Bloom filter
- HBASE-4532: Added a separate ROW-only Bloom filter for DeleteFamily
- HBASE-4585: Seek on deleted KV
- HBASE-4962: Top-of-the-column seek
So, you can see that you should upgrade your HBase installation. CDH 4.2
is out, and CDH 5.0 will be available very soon.[2]
[1] http://www.slideshare.net/cloudera/tag/hbasecon-2012
[2]
http://blog.cloudera.com/blog/2012/11/apache-hbase-assignmentmanager-improvements/
>
> 2013-02-16 05:30:45,978 [Thread-158] WARN
> org.apache.hadoop.mapred.LocalJobRunner - job_local_0005
> org.apache.hadoop.hbase.client.ScannerTimeoutException: 614527ms passed
> since the last invocation, timeout is currently set to 60000
> at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1302)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:133)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:142)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423)
> at
> org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> Caused by: org.apache.hadoop.hbase.UnknownScannerException:
> org.apache.hadoop.hbase.UnknownScannerException: Name: -5689607602334883791
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2110)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:84)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
Marcos Ortiz Valmaseda,
Product Manager && Data Scientist at UCI
Blog: http://marcosluis2186.posterous.com
Twitter: @marcosluis2186 <http://twitter.com/marcosluis2186>
