Hi,

We recently encountered the issue that HBase tables got into a state that was not disabled nor enabled. We found that the root cause was the linux clock skewed more than 5 hours. I googled and understood that hbase can only  handle about a couple of seconds time skew. We were wondering if there's any configuration in HBase that we can do so as to increase the number of seconds that hbase could handle?

Thanks very much,

YuLing

Are you using 0.94.x ?

If so, see the following:

    maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
    warningSkew = c.getLong("hbase.master.warningclockskew", 10000);
./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java

Cheers

On Fri, Mar 15, 2013 at 3:49 PM, <[EMAIL PROTECTED]> wrote:

> Hi,
>
> We recently encountered the issue that HBase tables got into a state that
> was not disabled nor enabled. We found that the root cause was the linux
> clock skewed more than 5 hours. I googled and understood that hbase can
> only  handle about a couple of seconds time skew. We were wondering if
> there's any configuration in HBase that we can do so as to increase the
> number of seconds that hbase could handle?
>
> Thanks very much,
>
> YuLing
>

Yes, I'm using 0.94.1.

-----Original Message-----
From: Ted Yu [mailto:[EMAIL PROTECTED]]
Sent: Friday, March 15, 2013 3:53 PM
To: [EMAIL PROTECTED]
Subject: Re: hbase time skew

Are you using 0.94.x ?

If so, see the following:

    maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
    warningSkew = c.getLong("hbase.master.warningclockskew", 10000); ./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java

Cheers

On Fri, Mar 15, 2013 at 3:49 PM, <[EMAIL PROTECTED]> wrote:

> Hi,
>
> We recently encountered the issue that HBase tables got into a state
> that was not disabled nor enabled. We found that the root cause was
> the linux clock skewed more than 5 hours. I googled and understood
> that hbase can only  handle about a couple of seconds time skew. We
> were wondering if there's any configuration in HBase that we can do so
> as to increase the number of seconds that hbase could handle?
>
> Thanks very much,
>
> YuLing
>

Please upgrade to 0.94.5, if possible.

There have been a lot of bug fixes and performance improvements since
0.94.1.

On Fri, Mar 15, 2013 at 4:26 PM, <[EMAIL PROTECTED]> wrote:

> Yes, I'm using 0.94.1.
>
> -----Original Message-----
> From: Ted Yu [mailto:[EMAIL PROTECTED]]
> Sent: Friday, March 15, 2013 3:53 PM
> To: [EMAIL PROTECTED]
> Subject: Re: hbase time skew
>
> Are you using 0.94.x ?
>
> If so, see the following:
>
>     maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
>     warningSkew = c.getLong("hbase.master.warningclockskew", 10000);
> ./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
>
> Cheers
>
> On Fri, Mar 15, 2013 at 3:49 PM, <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > We recently encountered the issue that HBase tables got into a state
> > that was not disabled nor enabled. We found that the root cause was
> > the linux clock skewed more than 5 hours. I googled and understood
> > that hbase can only  handle about a couple of seconds time skew. We
> > were wondering if there's any configuration in HBase that we can do so
> > as to increase the number of seconds that hbase could handle?
> >
> > Thanks very much,
> >
> > YuLing
> >
>

Hi Ted,

For the source code below, what's the unit of 30000? Is the maxSkew 30 seconds?  Also, what does Hbase do when the time skew exeeds 10 seconds and 30 seconds? Are these two values configurable?

Thanks,

YuLing
>maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
>     warningSkew = c.getLong("hbase.master.warningclockskew", 10000);
> ./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
-----Original Message-----
From: Ted Yu [mailto:[EMAIL PROTECTED]]
Sent: Friday, March 15, 2013 4:28 PM
To: [EMAIL PROTECTED]
Subject: Re: hbase time skew

Please upgrade to 0.94.5, if possible.

There have been a lot of bug fixes and performance improvements since 0.94.1.

On Fri, Mar 15, 2013 at 4:26 PM, <[EMAIL PROTECTED]> wrote:

> Yes, I'm using 0.94.1.
>
> -----Original Message-----
> From: Ted Yu [mailto:[EMAIL PROTECTED]]
> Sent: Friday, March 15, 2013 3:53 PM
> To: [EMAIL PROTECTED]
> Subject: Re: hbase time skew
>
> Are you using 0.94.x ?
>
> If so, see the following:
>
>     maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
>     warningSkew = c.getLong("hbase.master.warningclockskew", 10000);
> ./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
>
> Cheers
>
> On Fri, Mar 15, 2013 at 3:49 PM, <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > We recently encountered the issue that HBase tables got into a state
> > that was not disabled nor enabled. We found that the root cause was
> > the linux clock skewed more than 5 hours. I googled and understood
> > that hbase can only  handle about a couple of seconds time skew. We
> > were wondering if there's any configuration in HBase that we can do
> > so as to increase the number of seconds that hbase could handle?
> >
> > Thanks very much,
> >
> > YuLing
> >
>

When a region server reports to master the first time, ServerManager does
the following check:

   * Checks if the clock skew between the server and the master. If the
clock skew exceeds the

   * configured max, it will throw an exception; if it exceeds the
configured warning threshold,

   * it will log a warning but start normally.
The unit is in milliseconds.
These two parameters are configurable.

Please take a look at the source code if possible - the code should come
with tar ball.

Cheers

On Fri, Mar 15, 2013 at 4:50 PM, <[EMAIL PROTECTED]> wrote:

>  Hi Ted,
>
> For the source code below, what's the unit of 30000? Is the maxSkew 30
> seconds?  Also, what does Hbase do when the time skew exeeds 10 seconds and
> 30 seconds? Are these two values configurable?
>
> Thanks,
>
> YuLing
> >maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
> >     warningSkew = c.getLong("hbase.master.warningclockskew", 10000);
> > ./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
>
>
> -----Original Message-----
> From: Ted Yu [mailto:[EMAIL PROTECTED]]
> Sent: Friday, March 15, 2013 4:28 PM
> To: [EMAIL PROTECTED]
> Subject: Re: hbase time skew
>
> Please upgrade to 0.94.5, if possible.
>
> There have been a lot of bug fixes and performance improvements since
> 0.94.1.
>
> On Fri, Mar 15, 2013 at 4:26 PM, <[EMAIL PROTECTED]> wrote:
>
> > Yes, I'm using 0.94.1.
> >
> > -----Original Message-----
> > From: Ted Yu [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, March 15, 2013 3:53 PM
> > To: [EMAIL PROTECTED]
> > Subject: Re: hbase time skew
> >
> > Are you using 0.94.x ?
> >
> > If so, see the following:
> >
> >     maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
> >     warningSkew = c.getLong("hbase.master.warningclockskew", 10000);
> > ./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
> >
> > Cheers
> >
> > On Fri, Mar 15, 2013 at 3:49 PM, <[EMAIL PROTECTED]> wrote:
> >
> > > Hi,
> > >
> > > We recently encountered the issue that HBase tables got into a state
> > > that was not disabled nor enabled. We found that the root cause was
> > > the linux clock skewed more than 5 hours. I googled and understood
> > > that hbase can only  handle about a couple of seconds time skew. We
> > > were wondering if there's any configuration in HBase that we can do
> > > so as to increase the number of seconds that hbase could handle?
> > >
> > > Thanks very much,
> > >
> > > YuLing
> > >
> >
>

I just realized that if the clock on region server(s) drifts over
relatively long period of time (assuming region server stays up), the
following mechanism wouldn't work.

FYI

On Fri, Mar 15, 2013 at 5:07 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> When a region server reports to master the first time, ServerManager does
> the following check:
>
>    * Checks if the clock skew between the server and the master. If the
> clock skew exceeds the
>
>    * configured max, it will throw an exception; if it exceeds the
> configured warning threshold,
>
>    * it will log a warning but start normally.
> The unit is in milliseconds.
> These two parameters are configurable.
>
> Please take a look at the source code if possible - the code should come
> with tar ball.
>
> Cheers
>
> On Fri, Mar 15, 2013 at 4:50 PM, <[EMAIL PROTECTED]> wrote:
>
>>  Hi Ted,
>>
>> For the source code below, what's the unit of 30000? Is the maxSkew 30
>> seconds?  Also, what does Hbase do when the time skew exeeds 10 seconds and
>> 30 seconds? Are these two values configurable?
>>
>> Thanks,
>>
>> YuLing
>> >maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
>> >     warningSkew = c.getLong("hbase.master.warningclockskew", 10000);
>> > ./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
>>
>>
>> -----Original Message-----
>> From: Ted Yu [mailto:[EMAIL PROTECTED]]
>> Sent: Friday, March 15, 2013 4:28 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: hbase time skew
>>
>> Please upgrade to 0.94.5, if possible.
>>
>> There have been a lot of bug fixes and performance improvements since
>> 0.94.1.
>>
>> On Fri, Mar 15, 2013 at 4:26 PM, <[EMAIL PROTECTED]> wrote:
>>
>> > Yes, I'm using 0.94.1.
>> >
>> > -----Original Message-----
>> > From: Ted Yu [mailto:[EMAIL PROTECTED]]
>> > Sent: Friday, March 15, 2013 3:53 PM
>> > To: [EMAIL PROTECTED]
>> > Subject: Re: hbase time skew
>> >
>> > Are you using 0.94.x ?
>> >
>> > If so, see the following:
>> >
>> >     maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
>> >     warningSkew = c.getLong("hbase.master.warningclockskew", 10000);
>> > ./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
>> >
>> > Cheers
>> >
>> > On Fri, Mar 15, 2013 at 3:49 PM, <[EMAIL PROTECTED]> wrote:
>> >
>> > > Hi,
>> > >
>> > > We recently encountered the issue that HBase tables got into a state
>> > > that was not disabled nor enabled. We found that the root cause was
>> > > the linux clock skewed more than 5 hours. I googled and understood
>> > > that hbase can only  handle about a couple of seconds time skew. We
>> > > were wondering if there's any configuration in HBase that we can do
>> > > so as to increase the number of seconds that hbase could handle?
>> > >
>> > > Thanks very much,
>> > >
>> > > YuLing
>> > >
>> >
>>
>
>

Thanks very much Ted for the information. That's very helpful.

Just wondering if you have any idea where to configure these two values? I don't see these two parameters in any files under hbase/conf?

Thanks,

YuLing
-----Original Message-----
From: Ted Yu [mailto:[EMAIL PROTECTED]]
Sent: Friday, March 15, 2013 5:13 PM
To: [EMAIL PROTECTED]
Subject: Re: hbase time skew

I just realized that if the clock on region server(s) drifts over relatively long period of time (assuming region server stays up), the following mechanism wouldn't work.

FYI

On Fri, Mar 15, 2013 at 5:07 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> When a region server reports to master the first time, ServerManager
> does the following check:
>
>    * Checks if the clock skew between the server and the master. If
> the clock skew exceeds the
>
>    * configured max, it will throw an exception; if it exceeds the
> configured warning threshold,
>
>    * it will log a warning but start normally.
> The unit is in milliseconds.
> These two parameters are configurable.
>
> Please take a look at the source code if possible - the code should
> come with tar ball.
>
> Cheers
>
> On Fri, Mar 15, 2013 at 4:50 PM, <[EMAIL PROTECTED]> wrote:
>
>>  Hi Ted,
>>
>> For the source code below, what's the unit of 30000? Is the maxSkew
>> 30 seconds?  Also, what does Hbase do when the time skew exeeds 10
>> seconds and
>> 30 seconds? Are these two values configurable?
>>
>> Thanks,
>>
>> YuLing
>> >maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
>> >     warningSkew = c.getLong("hbase.master.warningclockskew",
>> >10000);  
>> >./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
>>
>>
>> -----Original Message-----
>> From: Ted Yu [mailto:[EMAIL PROTECTED]]
>> Sent: Friday, March 15, 2013 4:28 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: hbase time skew
>>
>> Please upgrade to 0.94.5, if possible.
>>
>> There have been a lot of bug fixes and performance improvements since
>> 0.94.1.
>>
>> On Fri, Mar 15, 2013 at 4:26 PM, <[EMAIL PROTECTED]> wrote:
>>
>> > Yes, I'm using 0.94.1.
>> >
>> > -----Original Message-----
>> > From: Ted Yu [mailto:[EMAIL PROTECTED]]
>> > Sent: Friday, March 15, 2013 3:53 PM
>> > To: [EMAIL PROTECTED]
>> > Subject: Re: hbase time skew
>> >
>> > Are you using 0.94.x ?
>> >
>> > If so, see the following:
>> >
>> >     maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
>> >     warningSkew = c.getLong("hbase.master.warningclockskew",
>> > 10000);
>> > ./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
>> >
>> > Cheers
>> >
>> > On Fri, Mar 15, 2013 at 3:49 PM, <[EMAIL PROTECTED]> wrote:
>> >
>> > > Hi,
>> > >
>> > > We recently encountered the issue that HBase tables got into a
>> > > state that was not disabled nor enabled. We found that the root
>> > > cause was the linux clock skewed more than 5 hours. I googled and
>> > > understood that hbase can only  handle about a couple of seconds
>> > > time skew. We were wondering if there's any configuration in
>> > > HBase that we can do so as to increase the number of seconds that hbase could handle?
>> > >
>> > > Thanks very much,
>> > >
>> > > YuLing
>> > >
>> >
>>
>
>

You can add them to your hbase-site.xml

Cheers

On Fri, Mar 15, 2013 at 5:24 PM, <[EMAIL PROTECTED]> wrote:

>
> Thanks very much Ted for the information. That's very helpful.
>
> Just wondering if you have any idea where to configure these two values? I
> don't see these two parameters in any files under hbase/conf?
>
> Thanks,
>
> YuLing
> -----Original Message-----
> From: Ted Yu [mailto:[EMAIL PROTECTED]]
> Sent: Friday, March 15, 2013 5:13 PM
> To: [EMAIL PROTECTED]
> Subject: Re: hbase time skew
>
> I just realized that if the clock on region server(s) drifts over
> relatively long period of time (assuming region server stays up), the
> following mechanism wouldn't work.
>
> FYI
>
> On Fri, Mar 15, 2013 at 5:07 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > When a region server reports to master the first time, ServerManager
> > does the following check:
> >
> >    * Checks if the clock skew between the server and the master. If
> > the clock skew exceeds the
> >
> >    * configured max, it will throw an exception; if it exceeds the
> > configured warning threshold,
> >
> >    * it will log a warning but start normally.
> > The unit is in milliseconds.
> > These two parameters are configurable.
> >
> > Please take a look at the source code if possible - the code should
> > come with tar ball.
> >
> > Cheers
> >
> > On Fri, Mar 15, 2013 at 4:50 PM, <[EMAIL PROTECTED]> wrote:
> >
> >>  Hi Ted,
> >>
> >> For the source code below, what's the unit of 30000? Is the maxSkew
> >> 30 seconds?  Also, what does Hbase do when the time skew exeeds 10
> >> seconds and
> >> 30 seconds? Are these two values configurable?
> >>
> >> Thanks,
> >>
> >> YuLing
> >> >maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
> >> >     warningSkew = c.getLong("hbase.master.warningclockskew",
> >> >10000);
> >> >./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
> >>
> >>
> >> -----Original Message-----
> >> From: Ted Yu [mailto:[EMAIL PROTECTED]]
> >> Sent: Friday, March 15, 2013 4:28 PM
> >> To: [EMAIL PROTECTED]
> >> Subject: Re: hbase time skew
> >>
> >> Please upgrade to 0.94.5, if possible.
> >>
> >> There have been a lot of bug fixes and performance improvements since
> >> 0.94.1.
> >>
> >> On Fri, Mar 15, 2013 at 4:26 PM, <[EMAIL PROTECTED]> wrote:
> >>
> >> > Yes, I'm using 0.94.1.
> >> >
> >> > -----Original Message-----
> >> > From: Ted Yu [mailto:[EMAIL PROTECTED]]
> >> > Sent: Friday, March 15, 2013 3:53 PM
> >> > To: [EMAIL PROTECTED]
> >> > Subject: Re: hbase time skew
> >> >
> >> > Are you using 0.94.x ?
> >> >
> >> > If so, see the following:
> >> >
> >> >     maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
> >> >     warningSkew = c.getLong("hbase.master.warningclockskew",
> >> > 10000);
> >> > ./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
> >> >
> >> > Cheers
> >> >
> >> > On Fri, Mar 15, 2013 at 3:49 PM, <[EMAIL PROTECTED]> wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > We recently encountered the issue that HBase tables got into a
> >> > > state that was not disabled nor enabled. We found that the root
> >> > > cause was the linux clock skewed more than 5 hours. I googled and
> >> > > understood that hbase can only  handle about a couple of seconds
> >> > > time skew. We were wondering if there's any configuration in
> >> > > HBase that we can do so as to increase the number of seconds that
> hbase could handle?
> >> > >
> >> > > Thanks very much,
> >> > >
> >> > > YuLing
> >> > >
> >> >
> >>
> >
> >
>

Create an ntp server local to the cluster? This will eliminate the skew in the first place.

Sent from a remote device. Please excuse any typos...

Mike Segel

On Mar 15, 2013, at 3:53 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> Are you using 0.94.x ?
>
> If so, see the following:
>
>    maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
>    warningSkew = c.getLong("hbase.master.warningclockskew", 10000);
> ./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
>
> Cheers
>
> On Fri, Mar 15, 2013 at 3:49 PM, <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> We recently encountered the issue that HBase tables got into a state that
>> was not disabled nor enabled. We found that the root cause was the linux
>> clock skewed more than 5 hours. I googled and understood that hbase can
>> only  handle about a couple of seconds time skew. We were wondering if
>> there's any configuration in HBase that we can do so as to increase the
>> number of seconds that hbase could handle?
>>
>> Thanks very much,
>>
>> YuLing
>>

I am going to agree with Michael on this one.  Don't change the clock
skew, fix it.

On Fri, Mar 15, 2013 at 9:56 PM, Michel Segel <[EMAIL PROTECTED]> wrote:
> Create an ntp server local to the cluster? This will eliminate the skew in the first place.
>
> Sent from a remote device. Please excuse any typos...
>
> Mike Segel
>
> On Mar 15, 2013, at 3:53 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
>> Are you using 0.94.x ?
>>
>> If so, see the following:
>>
>>    maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
>>    warningSkew = c.getLong("hbase.master.warningclockskew", 10000);
>> ./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
>>
>> Cheers
>>
>> On Fri, Mar 15, 2013 at 3:49 PM, <[EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>>
>>> We recently encountered the issue that HBase tables got into a state that
>>> was not disabled nor enabled. We found that the root cause was the linux
>>> clock skewed more than 5 hours. I googled and understood that hbase can
>>> only  handle about a couple of seconds time skew. We were wondering if
>>> there's any configuration in HBase that we can do so as to increase the
>>> number of seconds that hbase could handle?
>>>
>>> Thanks very much,
>>>
>>> YuLing
>>>

--
Kevin O'Dell
Customer Operations Engineer, Cloudera

The problem is that in our case, the customer configures the NTP server and it could be invalid. We're trying to cover user error cases, but on the other hand we're trying to understand how big time skew hbase can handle...

Thanks,

YuLing

-----Original Message-----
From: Kevin O'dell [mailto:[EMAIL PROTECTED]]
Sent: Saturday, March 16, 2013 5:49 AM
To: [EMAIL PROTECTED]
Subject: Re: hbase time skew

I am going to agree with Michael on this one.  Don't change the clock skew, fix it.

On Fri, Mar 15, 2013 at 9:56 PM, Michel Segel <[EMAIL PROTECTED]> wrote:
> Create an ntp server local to the cluster? This will eliminate the skew in the first place.
>
> Sent from a remote device. Please excuse any typos...
>
> Mike Segel
>
> On Mar 15, 2013, at 3:53 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
>> Are you using 0.94.x ?
>>
>> If so, see the following:
>>
>>    maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
>>    warningSkew = c.getLong("hbase.master.warningclockskew", 10000);
>> ./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
>>
>> Cheers
>>
>> On Fri, Mar 15, 2013 at 3:49 PM, <[EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>>
>>> We recently encountered the issue that HBase tables got into a state
>>> that was not disabled nor enabled. We found that the root cause was
>>> the linux clock skewed more than 5 hours. I googled and understood
>>> that hbase can only  handle about a couple of seconds time skew. We
>>> were wondering if there's any configuration in HBase that we can do
>>> so as to increase the number of seconds that hbase could handle?
>>>
>>> Thanks very much,
>>>
>>> YuLing
>>>

--
Kevin O'Dell
Customer Operations Engineer, Cloudera

The actual time isn't an issue. It's that all of the nodes in the cluster have the same time...
Give or take a couple of ms.

Sent from a remote device. Please excuse any typos...

Mike Segel

On Mar 18, 2013, at 2:39 PM, <[EMAIL PROTECTED]> wrote:

> The problem is that in our case, the customer configures the NTP server and it could be invalid. We're trying to cover user error cases, but on the other hand we're trying to understand how big time skew hbase can handle...
>
> Thanks,
>
> YuLing
>
> -----Original Message-----
> From: Kevin O'dell [mailto:[EMAIL PROTECTED]]
> Sent: Saturday, March 16, 2013 5:49 AM
> To: [EMAIL PROTECTED]
> Subject: Re: hbase time skew
>
> I am going to agree with Michael on this one.  Don't change the clock skew, fix it.
>
> On Fri, Mar 15, 2013 at 9:56 PM, Michel Segel <[EMAIL PROTECTED]> wrote:
>> Create an ntp server local to the cluster? This will eliminate the skew in the first place.
>>
>> Sent from a remote device. Please excuse any typos...
>>
>> Mike Segel
>>
>> On Mar 15, 2013, at 3:53 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>
>>> Are you using 0.94.x ?
>>>
>>> If so, see the following:
>>>
>>>   maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
>>>   warningSkew = c.getLong("hbase.master.warningclockskew", 10000);
>>> ./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
>>>
>>> Cheers
>>>
>>> On Fri, Mar 15, 2013 at 3:49 PM, <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi,
>>>>
>>>> We recently encountered the issue that HBase tables got into a state
>>>> that was not disabled nor enabled. We found that the root cause was
>>>> the linux clock skewed more than 5 hours. I googled and understood
>>>> that hbase can only  handle about a couple of seconds time skew. We
>>>> were wondering if there's any configuration in HBase that we can do
>>>> so as to increase the number of seconds that hbase could handle?
>>>>
>>>> Thanks very much,
>>>>
>>>> YuLing
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
>

So, how about the time changes? Actually in our case, we only have one node.   Let's say initially the customer does not install a NTP server and the clock becomes slow. Then the customer installs a new NTP server, the clock is adjusted by the NTP server, which ends up being advanced 5 mins.    Would this 'sudden' clock change cause any issues?  We tested on our test bed and found 5 hours advance of the clock would cause major issues. I'm not sure if it would be the same behavior if the clock is being advanced 3 ~ 5 mins?

Thanks,

YuLing

-----Original Message-----
From: Michel Segel [mailto:[EMAIL PROTECTED]]
Sent: Monday, March 18, 2013 2:29 PM
To: [EMAIL PROTECTED]
Subject: Re: hbase time skew

The actual time isn't an issue. It's that all of the nodes in the cluster have the same time...
Give or take a couple of ms.

Sent from a remote device. Please excuse any typos...

Mike Segel

On Mar 18, 2013, at 2:39 PM, <[EMAIL PROTECTED]> wrote:

> The problem is that in our case, the customer configures the NTP server and it could be invalid. We're trying to cover user error cases, but on the other hand we're trying to understand how big time skew hbase can handle...
>
> Thanks,
>
> YuLing
>
> -----Original Message-----
> From: Kevin O'dell [mailto:[EMAIL PROTECTED]]
> Sent: Saturday, March 16, 2013 5:49 AM
> To: [EMAIL PROTECTED]
> Subject: Re: hbase time skew
>
> I am going to agree with Michael on this one.  Don't change the clock skew, fix it.
>
> On Fri, Mar 15, 2013 at 9:56 PM, Michel Segel <[EMAIL PROTECTED]> wrote:
>> Create an ntp server local to the cluster? This will eliminate the skew in the first place.
>>
>> Sent from a remote device. Please excuse any typos...
>>
>> Mike Segel
>>
>> On Mar 15, 2013, at 3:53 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>
>>> Are you using 0.94.x ?
>>>
>>> If so, see the following:
>>>
>>>   maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
>>>   warningSkew = c.getLong("hbase.master.warningclockskew", 10000);
>>> ./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
>>>
>>> Cheers
>>>
>>> On Fri, Mar 15, 2013 at 3:49 PM, <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi,
>>>>
>>>> We recently encountered the issue that HBase tables got into a
>>>> state that was not disabled nor enabled. We found that the root
>>>> cause was the linux clock skewed more than 5 hours. I googled and
>>>> understood that hbase can only  handle about a couple of seconds
>>>> time skew. We were wondering if there's any configuration in HBase
>>>> that we can do so as to increase the number of seconds that hbase could handle?
>>>>
>>>> Thanks very much,
>>>>
>>>> YuLing
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
>