Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> how to get the time of a hadoop cluster, v0.20.2


Copy link to this message
-
Re: how to get the time of a hadoop cluster, v0.20.2
Uhm... sort of...

Niels is essentially correct and for the most of us, just starting an NNTPd on a server that sync's with a government clock and then your local servers sync to that... will be enough. However... in more detail...

Time is relative. ;-)

Ok... being a bit more serious...

There are two things you have to consider... What is meant by 'cluster time'?  and What you want to achieve?

Each machine in the cluster has its own clock. These will still have a certain amount of drift throughout the day.

So you can set up your own NTP server. (You can either run NTPd and sync to a known government clock) or you can spend money and buy an atomic clock for your servers or machine room.
(See http://www.atomic-clock.galleon.eu.com/ )

Then periodically throughout the day, via cron, have the machines in your machine room sync to the local NTP server.
This way all of your machines will have the same and correct time.

So this will sync the clocks to a degree, but then drift sets in.

Of course you also need to set up a machine to sync from... my vote would be the Name node. ;-)

HTH

-Mike
On May 16, 2013, at 10:34 AM, Niels Basjes <[EMAIL PROTECTED]> wrote:

> If you make sure that everything uses NTP then this becomes an irrelevant
> distinction.
>
>
> On Thu, May 16, 2013 at 4:01 PM, Jane Wayne <[EMAIL PROTECTED]>wrote:
>
>> yes, but that gets the current time on the server, not the hadoop cluster.
>> i need to be able to probe the date/time of the hadoop cluster.
>>
>>
>> On Tue, May 14, 2013 at 5:09 PM, Niels Basjes <[EMAIL PROTECTED]> wrote:
>>
>>> I made a typo. I meant API (instead of SPI).
>>>
>>> Have a look at this for more information:
>>>
>>>
>> http://stackoverflow.com/questions/833768/java-code-for-getting-current-time
>>>
>>>
>>> If you have a client that is not under NTP then that should be the way to
>>> fix your issue.
>>> Once you  have that getting the current time is easy.
>>>
>>> Niels Basjes
>>>
>>>
>>>
>>> On Tue, May 14, 2013 at 5:46 PM, Jane Wayne <[EMAIL PROTECTED]
>>>> wrote:
>>>
>>>> niels,
>>>>
>>>> i'm not familiar with the native java spi. spi = service provider
>>>> interface? could you let me know if this spi is part of the hadoop
>>>> api? if so, which package/class?
>>>>
>>>> but yes, all nodes on the cluster are using NTP to synchronize time.
>>>> however, the server (which is not a part of the hadoop cluster)
>>>> accessing/interfacing with the hadoop cluster cannot be assumed to be
>>>> using NTP. will this still make a difference? and actually, this is
>>>> the primary reason why i need to get the date/time of the hadoop
>>>> cluster (need to check if the date/time of the hadooop cluster is in
>>>> sync with the server).
>>>>
>>>>
>>>>
>>>> On Tue, May 14, 2013 at 11:38 AM, Niels Basjes <[EMAIL PROTECTED]>
>> wrote:
>>>>> If you have all nodes using NTP then you can simply use the native
>> Java
>>>> SPI
>>>>> to get the current system time.
>>>>>
>>>>>
>>>>> On Tue, May 14, 2013 at 4:41 PM, Jane Wayne <
>> [EMAIL PROTECTED]
>>>>> wrote:
>>>>>
>>>>>> hi all,
>>>>>>
>>>>>> is there a way to get the current time of a hadoop cluster via the
>>>>>> api? in particular, getting the time from the namenode or jobtracker
>>>>>> would suffice.
>>>>>>
>>>>>> i looked at JobClient but didn't see anything helpful.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards / Met vriendelijke groeten,
>>>>>
>>>>> Niels Basjes
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards / Met vriendelijke groeten,
>>>
>>> Niels Basjes
>>>
>>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB