Hello,

I'm getting scanner lease exceptions during mapreduce job [1] after running
it for less than 7 minutes. Though I have set
hbase.regionserver.lease.period to 600000 (i.e. 10 min) in hbase
configuration on master and all regionservers and master (and restarted
all). Also set it in job's configuration.

Is there anything else I could do to eliminate the exception?

Thanks,
Alex

[1]

 org.apache.hadoop.hbase.regionserver.LeaseException:
org.apache.hadoop.hbase.regionserver.LeaseException: lease
ʼ3854333525977315054ʼ does not exist
                  at
org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)

                  at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1862)

                  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
Source)

                  at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
Last
                  at
java.lang.reflect.Method.invoke(Method.java:597)
4KB
al FAILED         at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
Last
                  at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
8KB

All
                  at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)

                  at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)

                  at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)

                  at
java.lang.reflect.Constructor.newInstance(Constructor.java:513)

                  at
org.ap

Which version of hbase are you using ?

Thanks

On Feb 12, 2012, at 10:41 AM, Alex Baranau <[EMAIL PROTECTED]> wrote:

> Hello,
>
> I'm getting scanner lease exceptions during mapreduce job [1] after running
> it for less than 7 minutes. Though I have set
> hbase.regionserver.lease.period to 600000 (i.e. 10 min) in hbase
> configuration on master and all regionservers and master (and restarted
> all). Also set it in job's configuration.
>
> Is there anything else I could do to eliminate the exception?
>
> Thanks,
> Alex
>
> [1]
>
> org.apache.hadoop.hbase.regionserver.LeaseException:
> org.apache.hadoop.hbase.regionserver.LeaseException: lease
> ʼ3854333525977315054ʼ does not exist
>                  at
> org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)
>
>                  at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1862)
>
>                  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
> Source)
>
>                  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> Last
>                  at
> java.lang.reflect.Method.invoke(Method.java:597)
> 4KB
> al FAILED         at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Last
>                  at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> 8KB
>
> All
>                  at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
>                  at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>
>                  at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>
>                  at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>
>                  at
> org.ap

Hi,

0.90.4-cdh3u2

Alex

On Sun, Feb 12, 2012 at 1:44 PM, <[EMAIL PROTECTED]> wrote:

> Which version of hbase are you using ?
>
> Thanks
>
>
>
> On Feb 12, 2012, at 10:41 AM, Alex Baranau <[EMAIL PROTECTED]>
> wrote:
>
> > Hello,
> >
> > I'm getting scanner lease exceptions during mapreduce job [1] after
> running
> > it for less than 7 minutes. Though I have set
> > hbase.regionserver.lease.period to 600000 (i.e. 10 min) in hbase
> > configuration on master and all regionservers and master (and restarted
> > all). Also set it in job's configuration.
> >
> > Is there anything else I could do to eliminate the exception?
> >
> > Thanks,
> > Alex
> >
> > [1]
> >
> > org.apache.hadoop.hbase.regionserver.LeaseException:
> > org.apache.hadoop.hbase.regionserver.LeaseException: lease
> > ʼ3854333525977315054ʼ does not exist
> >                  at
> > org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)
> >
> >                  at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1862)
> >
> >                  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
> > Source)
> >
> >                  at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > Last
> >                  at
> > java.lang.reflect.Method.invoke(Method.java:597)
> > 4KB
> > al FAILED         at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> > Last
> >                  at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> > 8KB
> >
> > All
> >                  at
> > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> >
> >                  at
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >
> >                  at
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >
> >                  at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >
> >                  at
> > org.ap
>

Hm, weird. Looks like now it goes well. What I did is once again restarted
several regionservers (noticed that some where holding regions which were
stuck in "in transition" state). And started back the job.

Btw, forgot to mention that caching on Scanner was set to only 64 and I the
estimate for records size is < 100KB. Anyhow, the weird thing was that the
limits were set to 10min and it started to fail even before 10 min has
passed (sometimes after 3-4 minutes).

Have to go and can't watch the job to the end, will write whether it
succeed or not. If face same problems again will try brutal variant: set
caching = 10 (or even 1), set batch = 10 (or even 1).

Alex

On Sun, Feb 12, 2012 at 1:49 PM, Alex Baranau <[EMAIL PROTECTED]>wrote:

> Hi,
>
> 0.90.4-cdh3u2
>
> Alex
>
>
> On Sun, Feb 12, 2012 at 1:44 PM, <[EMAIL PROTECTED]> wrote:
>
>> Which version of hbase are you using ?
>>
>> Thanks
>>
>>
>>
>> On Feb 12, 2012, at 10:41 AM, Alex Baranau <[EMAIL PROTECTED]>
>> wrote:
>>
>> > Hello,
>> >
>> > I'm getting scanner lease exceptions during mapreduce job [1] after
>> running
>> > it for less than 7 minutes. Though I have set
>> > hbase.regionserver.lease.period to 600000 (i.e. 10 min) in hbase
>> > configuration on master and all regionservers and master (and restarted
>> > all). Also set it in job's configuration.
>> >
>> > Is there anything else I could do to eliminate the exception?
>> >
>> > Thanks,
>> > Alex
>> >
>> > [1]
>> >
>> > org.apache.hadoop.hbase.regionserver.LeaseException:
>> > org.apache.hadoop.hbase.regionserver.LeaseException: lease
>> > ʼ3854333525977315054ʼ does not exist
>> >                  at
>> > org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)
>> >
>> >                  at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1862)
>> >
>> >                  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
>> > Source)
>> >
>> >                  at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Last
>> >                  at
>> > java.lang.reflect.Method.invoke(Method.java:597)
>> > 4KB
>> > al FAILED         at
>> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> > Last
>> >                  at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>> > 8KB
>> >
>> > All
>> >                  at
>> > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> > Method)
>> >
>> >                  at
>> >
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> >
>> >                  at
>> >
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> >
>> >                  at
>> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> >
>> >                  at
>> > org.ap
>>
>
>

I had same wired problem last week when i upgraded from cdh3u0 to cdh3u2.
I got lease exception even when the  hbase.regionserver.lease.period was
set '300000'. The lease occurred after 1 mins or more.
Prior the upgrade everything was fine, i was executing with map reduce 8
slots.
The lease totally disappeared when i use only 1 slot, i thought there were
some concurrency problem with MR integration, but this weekend some of the
machines where restarted and suddenly everything is fine!! I'm back to 8
slot and the extract are running. (i run several time even without even one
lease exception)
In my case NO region server were stuck in transition.
It is really frustrating that i cannot point on what was the real problem.
Even log with debug did not point on problems (perhaps because it is also
missing some debug statement like when a scanner lease is added to the RS)

Mikael.S
On Sun, Feb 12, 2012 at 9:55 PM, Alex Baranau <[EMAIL PROTECTED]>wrote:

> Hm, weird. Looks like now it goes well. What I did is once again restarted
> several regionservers (noticed that some where holding regions which were
> stuck in "in transition" state). And started back the job.
>
> Btw, forgot to mention that caching on Scanner was set to only 64 and I the
> estimate for records size is < 100KB. Anyhow, the weird thing was that the
> limits were set to 10min and it started to fail even before 10 min has
> passed (sometimes after 3-4 minutes).
>
> Have to go and can't watch the job to the end, will write whether it
> succeed or not. If face same problems again will try brutal variant: set
> caching = 10 (or even 1), set batch = 10 (or even 1).
>
> Alex
>
> On Sun, Feb 12, 2012 at 1:49 PM, Alex Baranau <[EMAIL PROTECTED]
> >wrote:
>
> > Hi,
> >
> > 0.90.4-cdh3u2
> >
> > Alex
> >
> >
> > On Sun, Feb 12, 2012 at 1:44 PM, <[EMAIL PROTECTED]> wrote:
> >
> >> Which version of hbase are you using ?
> >>
> >> Thanks
> >>
> >>
> >>
> >> On Feb 12, 2012, at 10:41 AM, Alex Baranau <[EMAIL PROTECTED]>
> >> wrote:
> >>
> >> > Hello,
> >> >
> >> > I'm getting scanner lease exceptions during mapreduce job [1] after
> >> running
> >> > it for less than 7 minutes. Though I have set
> >> > hbase.regionserver.lease.period to 600000 (i.e. 10 min) in hbase
> >> > configuration on master and all regionservers and master (and
> restarted
> >> > all). Also set it in job's configuration.
> >> >
> >> > Is there anything else I could do to eliminate the exception?
> >> >
> >> > Thanks,
> >> > Alex
> >> >
> >> > [1]
> >> >
> >> > org.apache.hadoop.hbase.regionserver.LeaseException:
> >> > org.apache.hadoop.hbase.regionserver.LeaseException: lease
> >> > ʼ3854333525977315054ʼ does not exist
> >> >                  at
> >> >
> org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)
> >> >
> >> >                  at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1862)
> >> >
> >> >                  at
> sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
> >> > Source)
> >> >
> >> >                  at
> >> >
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > Last
> >> >                  at
> >> > java.lang.reflect.Method.invoke(Method.java:597)
> >> > 4KB
> >> > al FAILED         at
> >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> >> > Last
> >> >                  at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> >> > 8KB
> >> >
> >> > All
> >> >                  at
> >> > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >> > Method)
> >> >
> >> >                  at
> >> >
> >>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >> >
> >> >                  at
> >> >
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >> >
> >> >                  at
> >> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)

Mikael.S

Yeah, looks like the same thing.

The job went well eventually (many mappers, lasted for 13h +, no errors at
all).

Alex

On Sun, Feb 12, 2012 at 4:35 PM, Mikael Sitruk <[EMAIL PROTECTED]>wrote:

> I had same wired problem last week when i upgraded from cdh3u0 to cdh3u2.
> I got lease exception even when the  hbase.regionserver.lease.period was
> set '300000'. The lease occurred after 1 mins or more.
> Prior the upgrade everything was fine, i was executing with map reduce 8
> slots.
> The lease totally disappeared when i use only 1 slot, i thought there were
> some concurrency problem with MR integration, but this weekend some of the
> machines where restarted and suddenly everything is fine!! I'm back to 8
> slot and the extract are running. (i run several time even without even one
> lease exception)
> In my case NO region server were stuck in transition.
> It is really frustrating that i cannot point on what was the real problem.
> Even log with debug did not point on problems (perhaps because it is also
> missing some debug statement like when a scanner lease is added to the RS)
>
> Mikael.S
>
>
> On Sun, Feb 12, 2012 at 9:55 PM, Alex Baranau <[EMAIL PROTECTED]
> >wrote:
>
> > Hm, weird. Looks like now it goes well. What I did is once again
> restarted
> > several regionservers (noticed that some where holding regions which were
> > stuck in "in transition" state). And started back the job.
> >
> > Btw, forgot to mention that caching on Scanner was set to only 64 and I
> the
> > estimate for records size is < 100KB. Anyhow, the weird thing was that
> the
> > limits were set to 10min and it started to fail even before 10 min has
> > passed (sometimes after 3-4 minutes).
> >
> > Have to go and can't watch the job to the end, will write whether it
> > succeed or not. If face same problems again will try brutal variant: set
> > caching = 10 (or even 1), set batch = 10 (or even 1).
> >
> > Alex
> >
> > On Sun, Feb 12, 2012 at 1:49 PM, Alex Baranau <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi,
> > >
> > > 0.90.4-cdh3u2
> > >
> > > Alex
> > >
> > >
> > > On Sun, Feb 12, 2012 at 1:44 PM, <[EMAIL PROTECTED]> wrote:
> > >
> > >> Which version of hbase are you using ?
> > >>
> > >> Thanks
> > >>
> > >>
> > >>
> > >> On Feb 12, 2012, at 10:41 AM, Alex Baranau <[EMAIL PROTECTED]>
> > >> wrote:
> > >>
> > >> > Hello,
> > >> >
> > >> > I'm getting scanner lease exceptions during mapreduce job [1] after
> > >> running
> > >> > it for less than 7 minutes. Though I have set
> > >> > hbase.regionserver.lease.period to 600000 (i.e. 10 min) in hbase
> > >> > configuration on master and all regionservers and master (and
> > restarted
> > >> > all). Also set it in job's configuration.
> > >> >
> > >> > Is there anything else I could do to eliminate the exception?
> > >> >
> > >> > Thanks,
> > >> > Alex
> > >> >
> > >> > [1]
> > >> >
> > >> > org.apache.hadoop.hbase.regionserver.LeaseException:
> > >> > org.apache.hadoop.hbase.regionserver.LeaseException: lease
> > >> > ʼ3854333525977315054ʼ does not exist
> > >> >                  at
> > >> >
> > org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)
> > >> >
> > >> >                  at
> > >> >
> > >>
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1862)
> > >> >
> > >> >                  at
> > sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
> > >> > Source)
> > >> >
> > >> >                  at
> > >> >
> > >>
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > >> > Last
> > >> >                  at
> > >> > java.lang.reflect.Method.invoke(Method.java:597)
> > >> > 4KB
> > >> > al FAILED         at
> > >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> > >> > Last
> > >> >                  at
> > >> >
> > >>
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> > >> > 8KB
> > >> >
> > >> > All
> > >> >                  at

Hi Alex
Do you use a filter for KVs in scanner? How many columns per row can you
have?
Thanks
Daniel

On 02/12/2012 09:55 PM, Alex Baranau wrote:
> forgot to mention that caching on Scanner was set to only 64 and I the
> estimate for records size is<  100KB