Hi.

Just wondering - does anyone know what framework Hadoop uses for
daemonizing?

Any chance it's jsvc from Apache?

Regards.

Hi Stas,

Hadoop doesn't daemonize itself. The shell scripts use nohup and a lot
of bash code to achieve a similar idea.

-Todd

On Thu, Feb 4, 2010 at 1:03 PM, Stas Oskin <[EMAIL PROTECTED]> wrote:
> Hi.
>
> Just wondering - does anyone know what framework Hadoop uses for
> daemonizing?
>
> Any chance it's jsvc from Apache?
>
> Regards.
>

Hi Todd.

Hadoop doesn't daemonize itself. The shell scripts use nohup and a lot
> of bash code to achieve a similar idea.
>
>
Was there any design decision behind this approach?

I remember that I had to do the same, as any wrapper just caused the daemon
to run in lower priority then it should.

Regards.

On 2/4/10 1:21 PM, "Stas Oskin" <[EMAIL PROTECTED]> wrote:

> Was there any design decision behind this approach?

Likely KISS.

> I remember that I had to do the same, as any wrapper just caused the daemon
> to run in lower priority then it should.

...which is also easily dealt with from the shell and gives you the
flexibility to use OS-specific constructs.

The other big benefit is that this also means you don't need to UNdaemonize
code for those users that use something besides just pure init rc scripts.
(djbtools, smf, launchd, whatever)

On Thu, Feb 4, 2010 at 1:21 PM, Stas Oskin <[EMAIL PROTECTED]> wrote:
> Hi Todd.
>
> Hadoop doesn't daemonize itself. The shell scripts use nohup and a lot
>> of bash code to achieve a similar idea.
>>
>>
> Was there any design decision behind this approach?
>

It long predates my involvement in the project. In fact, it predates
Hadoop itself - it got inherited from Nutch long ago.

I vaguely recall a JIRA about using jsvc for Hadoop - if you search
around I bet you can turn it up.

-Todd

I actually asked this because I'm looking for a good alternative to current
bunch of scripts and lsb-redhat dependencies I have today in my own Hadoop
client which runs as daemon. So I kinda hoped there is some "sauce" behind
Hadoop I can borrow.

While this might be not the most appropriate list, I'd appreciate if someone
can say if jsvc can keep the right priorities, or suggest alternative daemon
framework.

Thanks again.
It long predates my involvement in the project. In fact, it predates
> Hadoop itself - it got inherited from Nutch long ago.
>
> I vaguely recall a JIRA about using jsvc for Hadoop - if you search
> around I bet you can turn it up.
>
> -Todd
>

On Thu, Feb 4, 2010 at 4:39 PM, Stas Oskin <[EMAIL PROTECTED]> wrote:
> I actually asked this because I'm looking for a good alternative to current
> bunch of scripts and lsb-redhat dependencies I have today in my own Hadoop
> client which runs as daemon. So I kinda hoped there is some "sauce" behind
> Hadoop I can borrow.
>
> While this might be not the most appropriate list, I'd appreciate if someone
> can say if jsvc can keep the right priorities, or suggest alternative daemon
> framework.
>
> Thanks again.
>
>
> It long predates my involvement in the project. In fact, it predates
>> Hadoop itself - it got inherited from Nutch long ago.
>>
>> I vaguely recall a JIRA about using jsvc for Hadoop - if you search
>> around I bet you can turn it up.
>>
>> -Todd
>>
>
Stas,
Demonizing is one of those native bits java does not do well with by
default. jsrv is an option. I have never had a problem with nohup as
you have, although it is a bit hackish.

Some concepts I was considering
1) Deamontools - manages processes run in the foreground (handles
restarts), no need to demonize
2) linux-ha - much like init scripts but fancy cluster management capabilities

Personally, I am pretty happy with the cloudera LSB scripts. Missing
'status' but ps -ef or jps deals with that.

Do you just have general problems with 'nohup' or have you unearthed a
specific hadoop nohup issue?

Hi Edward.

Do you just have general problems with 'nohup' or have you unearthed a
> specific hadoop nohup issue?
>

Just to clarify that we speak about my own Java Hadoop connector here, not
about Hadoop itself, which works just great (with some added pepper from
monit for potential crashes).

I don't like the fact that for simple init script I need to add a full
redhat-lsb package, which just involves having a lot of packages installed.

If Cloudera LSB scripts are self-contained, and (most important) can
generate PID files, I will be happy to give them a look.

Thanks.

Hi,

these are the most used tools

- JSVC http://commons.apache.org/daemon/jsvc.html
<http://commons.apache.org/daemon/jsvc.html>- Java service wrapper
http://wrapper.tanukisoftware.org/

Windows only:
- JNA windows service
<http://wrapper.tanukisoftware.org/doc/english/download.jsp>- Windows
service wrapper
http://weblogs.java.net/blog/2008/09/29/winsw-windows-service-wrapper-less-restrictive-license

<http://weblogs.java.net/blog/2008/09/29/winsw-windows-service-wrapper-less-restrictive-license>
Regards,
Leen

On Thu, Feb 4, 2010 at 10:39 PM, Stas Oskin <[EMAIL PROTECTED]> wrote:

> I actually asked this because I'm looking for a good alternative to current
> bunch of scripts and lsb-redhat dependencies I have today in my own Hadoop
> client which runs as daemon. So I kinda hoped there is some "sauce" behind
> Hadoop I can borrow.
>
> While this might be not the most appropriate list, I'd appreciate if
> someone
> can say if jsvc can keep the right priorities, or suggest alternative
> daemon
> framework.
>
> Thanks again.
>
>
> It long predates my involvement in the project. In fact, it predates
> > Hadoop itself - it got inherited from Nutch long ago.
> >
> > I vaguely recall a JIRA about using jsvc for Hadoop - if you search
> > around I bet you can turn it up.
> >
> > -Todd
> >
>

Hi,

I'm working on a hadoop package for Debian, which also includes init scripts
using the daemon program (Debian package "daemon") from
http://www.libslack.org/daemon

You can have a look at the init script(s) at

http://git.debian.org/?p=pkg-java/hadoop.git;a=blob;f=debian/common-init.sh
http://git.debian.org/?p=pkg-java/hadoop.git;a=blob;f=debian/hadoop-wrapper.sh

The scripts still need some polishing though. The common-init.sh is sourced by
different init stubs which provide only the variable NAME, see for example:

http://git.debian.org/?p=pkg-java/hadoop.git;a=blob;f=debian/hadoop-
jobtrackerd.init

Best regards,

Thomas Koch, http://www.koch.ro

> > I'm working on a hadoop package for Debian, which also includes init
> > scripts
> > using the daemon program (Debian package "daemon") from
> > http://www.libslack.org/daemon
>
> Can these scripts be used on other distributions, like Red Hat? Or it's a
> Debian only daemon?
I'm not familiar enough with Red Hat do answer this. The first thing that
comes to my mind is, whether Red Hat has the same lsb_* scripts that I do
source.

> I'm actually used so far the Debian start-stop-daemon binary - maybe this
>  is the same?
No. start-stop-daemon is good for programs that behave as nice daemons
themselves. daemon is especially good for the ugly rest in that it:

- forwards stdout and stderr to a log of your wish instead of nowhere, like
  the standard hadoop script does

- not only writes pidfiles but also deletes them, when the daemonized process
  dies

- let's you easily check, whether a daemon is running, which is especially
  compliate with java stuff, because the executable is always "java"

There are some more nice features, which I've not used yet. Just try to get
the package "daemon" for your distribution and look up man 1 daemon.

Best regards,

Thomas Koch, http://www.koch.ro

Thomas Koch wrote:
>>> I'm working on a hadoop package for Debian, which also includes init
>>> scripts
>>> using the daemon program (Debian package "daemon") from
>>> http://www.libslack.org/daemon
>> Can these scripts be used on other distributions, like Red Hat? Or it's a
>> Debian only daemon?
> I'm not familiar enough with Red Hat do answer this. The first thing that
> comes to my mind is, whether Red Hat has the same lsb_* scripts that I do
> source.

RHEL doesn't, the  daemon scripts for SmartFrog have to look for both
RHEL and lsb; lsb is what you get on SuSE though.

From:
http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/release/scripts/etc/rc.d/init.d/smartfrogd?revision=8160&view=markup
if [ -f /lib/lsb/init-functions ]; then
   . /lib/lsb/init-functions
   STOP_SUPPORTED=0
   alias START_DAEMON=start_daemon
#  alias STOP_DAEMON=echo "ignoring stop daemon request"
   alias STATUS=MyStatus
   alias LOG_SUCCESS=log_success_msg
   alias LOG_FAILURE=log_failure_msg
   alias LOG_WARNING=log_warning_msg
elif [ -f /etc/init.d/functions ]; then
   . /etc/init.d/functions
   STOP_SUPPORTED=1
   alias START_DAEMON=start
   alias STOP_DAEMON=stop
   alias STATUS=status
   alias LOG_SUCCESS=success
   alias LOG_FAILURE=failure
   alias LOG_WARNING=passed
else
   echo "Error: your platform is not supported by $0" > /dev/stderr
   exit 1
fi

looks like I actually set a flag saying whether stopping the daemon
works reliably, hmmm.
Another handy bit of code:

java -version >/dev/null
if [ $? -ne 0 ]; then
   echo "Error: $0 cannot find java. Either it is not installed or the
PATH is incomplete" > /dev/stderr
exit 6
fi

Testing all of this is fun, I scp my RPMs to virtualized machines then
walk them through the lifecycle.

Hi.
> RHEL doesn't, the  daemon scripts for SmartFrog have to look for both RHEL
> and lsb; lsb is what you get on SuSE though.
>
>
Actually LSB is what I'm looking to distance from - too much dependencies
required for it on the RedHat for example.

I'm looking for a solution that won't rely much on external OS dependencies,
and will be basically self-sufficient.

Any ideas, other then a pure bash scripting?

Regards.

On Fri, Feb 12, 2010 at 1:39 PM, Stas Oskin <[EMAIL PROTECTED]> wrote:
> Hi.
>
>
>> RHEL doesn't, the  daemon scripts for SmartFrog have to look for both RHEL
>> and lsb; lsb is what you get on SuSE though.
>>
>>
> Actually LSB is what I'm looking to distance from - too much dependencies
> required for it on the RedHat for example.
>
> I'm looking for a solution that won't rely much on external OS dependencies,
> and will be basically self-sufficient.
>
> Any ideas, other then a pure bash scripting?
>
> Regards.
>

Stas,

The problem is only as simple or as hard as you want to make it. LSB
defines subs and return codes

http://refspecs.freestandards.org/LSB_3.1.1/LSB-Core-generic/LSB-Core-generic/iniscrptact.html

RH scripts source /etc/rc.d/init.d/functions, but you don't have to do that.

You said "pure bash", that is almost an oxymoron. You can not even
truly count on pidof, which, or /proc on all systems. Also the default
shell could be csh or sh.

Status is always the most interesting function, do you parse ps -ef,
do you write a pid file, do you nc to a port?

Personally, I would not lose sleep over shell scripts. After all the
fun part should be using hadoop, not starting it up.