Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> how to get the time of a hadoop cluster, v0.20.2


+
Jane Wayne 2013-05-14, 14:41
+
Niels Basjes 2013-05-14, 15:38
+
Jane Wayne 2013-05-14, 15:46
+
Niels Basjes 2013-05-14, 21:09
+
Jane Wayne 2013-05-16, 14:01
+
Niels Basjes 2013-05-16, 15:34
+
Michael Segel 2013-05-16, 16:46
+
Jane Wayne 2013-05-17, 11:51
+
Niels Basjes 2013-05-17, 13:17
+
Bertrand Dechoux 2013-05-17, 14:02
Copy link to this message
-
Re: how to get the time of a hadoop cluster, v0.20.2
"if NTP is correclty used"

that's the key statement. in several of our clusters, NTP setup is kludgy.
note that the professionals administering the cluster are different from
"us" the engineers. so, there's a lot of red tape to go through to get
something trivial or not fixed. we have noticed that NTP is not setup
correctly (using default GMT timezone, for example). without explaining all
the tedious details, this mismatch of date/time (of the hadoop cluster to
the server machine) is causing some pains.

i'm not sure i agree with "the local OS time from your server machine will
be the best estimation." that doesn't make sense.

but what i want to achieve is very simple. as stated before, i just want to
ask the namenode or jobtracker, "hey, what date/time do you have?"
unfortunately for me, as niels pointed out, this query is not possible via
the hadoop api.

thanks for helping, though.

:)
On Fri, May 17, 2013 at 10:02 AM, Bertrand Dechoux <[EMAIL PROTECTED]>wrote:

> For hadoop, 'cluster time' is the local OS time. You might want to get the
> time of the namenode machine but indeed if NTP is correctly used, the local
> OS time from your server machine will be the best estimation. If you
> request the time from the namenode machine, you will be penalized by the
> delay of your request.
>
> Regards
>
> Bertrand
>
>
> On Fri, May 17, 2013 at 3:17 PM, Niels Basjes <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > > i have another computer (which i have referred to as a server, since it
> > is
> > > running tomcat), and this computer is NOT a part of the hadoop cluster
> > (it
> > > doesn't run any of the hadoop daemons), but does submit jobs to the
> > hadoop
> > > cluster via a JEE webapp interface. i need to check that the time on
> this
> > > computer is in sync with the time on the hadoop cluster. when i say
> > "check
> > > that the time is in sync", there is a defined tolerance/threshold
> > > difference in date/time that i am willing to accept (e.g. the date/time
> > > should be the same down to the minute).
> >
> > If you ensure (using NTP) that all your servers have the same time then
> you
> > can simply query your local server for the time and you have the correct
> > answer to your question.
> >
> > You are searching for a solution in the Hadoop API (where this does not
> > exist) when the solution is present at a different level.
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>
+
Jane Wayne 2013-05-17, 14:13
+
Michael Segel 2013-05-18, 15:54
+
Jane Wayne 2013-05-17, 14:01