Hi guys,

In my deployment of Kafka I am using runit to control the service of the
kafka broker.

In the run script I have this line:
#!/bin/sh
exec 2>&1

exec /opt/kafka/bin/kafka-server-start.sh /etc/kafka/server.properties

It works fine, if I want to start the broker, but if I want to stop or
restart the process, the kafka broker doesn't get kill...

So eventually the old kafka broker prevents from the new broker to start
(Because the ports are already binded).
Of course the running the same line of code in the shell and then ctrl-c
ing works good (as you know kafka broker is not a daemon)

I was wondering if someone has encountered this phenomena and solved it.

Thanks,

Guy Doulberg

I think what you are saying is that you don't have the process id for the
broker so you don't have the right thing to kill?

Two solutions I know:
1. Change your script to store the process id somewhere, and use that for
the kill
2. Use pkill and just kill all kafka instances (only works if there is only
one broker per node)

-Jay

On Thu, Jul 5, 2012 at 2:24 AM, Guy Doulberg <[EMAIL PROTECTED]>wrote:

> Hi guys,
>
> In my deployment of Kafka I am using runit to control the service of the
> kafka broker.
>
> In the run script I have this line:
> #!/bin/sh
> exec 2>&1
>
> exec /opt/kafka/bin/kafka-server-**start.sh /etc/kafka/server.properties
>
>
>
> It works fine, if I want to start the broker, but if I want to stop or
> restart the process, the kafka broker doesn't get kill...
>
> So eventually the old kafka broker prevents from the new broker to start
> (Because the ports are already binded).
>
>
> Of course the running the same line of code in the shell and then ctrl-c
> ing works good (as you know kafka broker is not a daemon)
>
> I was wondering if someone has encountered this phenomena and solved it.
>
> Thanks,
>
> Guy Doulberg
>

Hi Jay,
Thanks for your replay.

Not exactly,

I can see that the pid the runit save, is deferent from the pid that I
acctually have running on the machine , could it be that the launcher of
the kafka broker forks another process with a different PID?

Thanks.

The only way I could kill the process is by using kill -9 , I suspect
that runit doesn't use the
On 07/05/2012 10:19 PM, Jay Kreps wrote:
> I think what you are saying is that you don't have the process id for the
> broker so you don't have the right thing to kill?
>
> Two solutions I know:
> 1. Change your script to store the process id somewhere, and use that for
> the kill
> 2. Use pkill and just kill all kafka instances (only works if there is only
> one broker per node)
>
> -Jay
>
> On Thu, Jul 5, 2012 at 2:24 AM, Guy Doulberg <[EMAIL PROTECTED]>wrote:
>
>> Hi guys,
>>
>> In my deployment of Kafka I am using runit to control the service of the
>> kafka broker.
>>
>> In the run script I have this line:
>> #!/bin/sh
>> exec 2>&1
>>
>> exec /opt/kafka/bin/kafka-server-**start.sh /etc/kafka/server.properties
>>
>>
>>
>> It works fine, if I want to start the broker, but if I want to stop or
>> restart the process, the kafka broker doesn't get kill...
>>
>> So eventually the old kafka broker prevents from the new broker to start
>> (Because the ports are already binded).
>>
>>
>> Of course the running the same line of code in the shell and then ctrl-c
>> ing works good (as you know kafka broker is not a daemon)
>>
>> I was wondering if someone has encountered this phenomena and solved it.
>>
>> Thanks,
>>
>> Guy Doulberg
>>

Acttully, I found out that there are 3 process that are created when
running:

/opt/kafka/bin/kafka-server-**start.sh /etc/kafka/server.properties

guyd      4883  4658  0 09:36 pts/5    00:00:00 /bin/bash ./kafka-server-start.sh ../config/server.properties
guyd      4885  4883  0 09:36 pts/5    00:00:00 /bin/bash ./kafka-run-class.sh kafka.Kafka ../config/server.properties
guyd      4887  4885  2 09:36 pts/5    00:00:00 java -Xmx512M -server -Dlog4j.configuration=file:./../config/log4j.properties -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9999 -cp :./../project/boot/scala-2.8.0/lib/scala-compiler.jar:./../project/boot/scala-2.8.0/lib/scala-library.jar:./../core/target/scala_2.8.0/kafka-0.7.0.jar:./../core/lib/zkclient-20110412.jar:./../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:./../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:./../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.3.jar kafka.Kafka ../config/server.properties
Why do we need these 3 processes?

On 07/08/2012 09:35 AM, Guy Doulberg wrote:
> Hi Jay,
> Thanks for your replay.
>
> Not exactly,
>
> I can see that the pid the runit save, is deferent from the pid that I
> acctually have running on the machine , could it be that the launcher of
> the kafka broker forks another process with a different PID?
>
> Thanks.
>
> The only way I could kill the process is by using kill -9 , I suspect
> that runit doesn't use the
> On 07/05/2012 10:19 PM, Jay Kreps wrote:
>> I think what you are saying is that you don't have the process id for the
>> broker so you don't have the right thing to kill?
>>
>> Two solutions I know:
>> 1. Change your script to store the process id somewhere, and use that for
>> the kill
>> 2. Use pkill and just kill all kafka instances (only works if there is only
>> one broker per node)
>>
>> -Jay
>>
>> On Thu, Jul 5, 2012 at 2:24 AM, Guy Doulberg <[EMAIL PROTECTED]>wrote:
>>
>>> Hi guys,
>>>
>>> In my deployment of Kafka I am using runit to control the service of the
>>> kafka broker.
>>>
>>> In the run script I have this line:
>>> #!/bin/sh
>>> exec 2>&1
>>>
>>> exec /opt/kafka/bin/kafka-server-**start.sh /etc/kafka/server.properties
>>>
>>>
>>>
>>> It works fine, if I want to start the broker, but if I want to stop or
>>> restart the process, the kafka broker doesn't get kill...
>>>
>>> So eventually the old kafka broker prevents from the new broker to start
>>> (Because the ports are already binded).
>>>
>>>
>>> Of course the running the same line of code in the shell and then ctrl-c
>>> ing works good (as you know kafka broker is not a daemon)
>>>
>>> I was wondering if someone has encountered this phenomena and solved it.
>>>
>>> Thanks,
>>>
>>> Guy Doulberg
>>>
>

The first 2 are just scripts calling the actual kafka code in jvm.

Thanks,

Jun

On Sun, Jul 8, 2012 at 12:12 AM, Guy Doulberg <[EMAIL PROTECTED]>wrote:

> Acttully, I found out that there are 3 process that are created when
> running:
>
> /opt/kafka/bin/kafka-server-****start.sh /etc/kafka/server.properties
>
> guyd      4883  4658  0 09:36 pts/5    00:00:00 /bin/bash
> ./kafka-server-start.sh ../config/server.properties
> guyd      4885  4883  0 09:36 pts/5    00:00:00 /bin/bash
> ./kafka-run-class.sh kafka.Kafka ../config/server.properties
> guyd      4887  4885  2 09:36 pts/5    00:00:00 java -Xmx512M -server
> -Dlog4j.configuration=file:./.**./config/log4j.properties
> -Dcom.sun.management.jmxremote -Dcom.sun.management.**jmxremote.authenticate=false
> -Dcom.sun.management.**jmxremote.ssl=false -Dcom.sun.management.**jmxremote.port=9999
> -cp :./../project/boot/scala-2.8.**0/lib/scala-compiler.jar:./../**
> project/boot/scala-2.8.0/lib/**scala-library.jar:./../core/**
> target/scala_2.8.0/kafka-0.7.**0.jar:./../core/lib/zkclient-**
> 20110412.jar:./../core/lib_**managed/scala_2.8.0/compile/**
> jopt-simple-3.2.jar:./../core/**lib_managed/scala_2.8.0/**
> compile/log4j-1.2.15.jar:./../**core/lib_managed/scala_2.8.0/**compile/zookeeper-3.3.3.jar
> kafka.Kafka ../config/server.properties
>
>
> Why do we need these 3 processes?
>
>
>
>
> On 07/08/2012 09:35 AM, Guy Doulberg wrote:
>
>> Hi Jay,
>> Thanks for your replay.
>>
>> Not exactly,
>>
>> I can see that the pid the runit save, is deferent from the pid that I
>> acctually have running on the machine , could it be that the launcher of
>> the kafka broker forks another process with a different PID?
>>
>> Thanks.
>>
>> The only way I could kill the process is by using kill -9 , I suspect
>> that runit doesn't use the
>> On 07/05/2012 10:19 PM, Jay Kreps wrote:
>>
>>> I think what you are saying is that you don't have the process id for the
>>> broker so you don't have the right thing to kill?
>>>
>>> Two solutions I know:
>>> 1. Change your script to store the process id somewhere, and use that for
>>> the kill
>>> 2. Use pkill and just kill all kafka instances (only works if there is
>>> only
>>> one broker per node)
>>>
>>> -Jay
>>>
>>> On Thu, Jul 5, 2012 at 2:24 AM, Guy Doulberg <[EMAIL PROTECTED]>*
>>> *wrote:
>>>
>>>  Hi guys,
>>>>
>>>> In my deployment of Kafka I am using runit to control the service of the
>>>> kafka broker.
>>>>
>>>> In the run script I have this line:
>>>> #!/bin/sh
>>>> exec 2>&1
>>>>
>>>> exec /opt/kafka/bin/kafka-server-****start.sh
>>>> /etc/kafka/server.properties
>>>>
>>>>
>>>>
>>>> It works fine, if I want to start the broker, but if I want to stop or
>>>> restart the process, the kafka broker doesn't get kill...
>>>>
>>>> So eventually the old kafka broker prevents from the new broker to start
>>>> (Because the ports are already binded).
>>>>
>>>>
>>>> Of course the running the same line of code in the shell and then ctrl-c
>>>> ing works good (as you know kafka broker is not a daemon)
>>>>
>>>> I was wondering if someone has encountered this phenomena and solved it.
>>>>
>>>> Thanks,
>>>>
>>>> Guy Doulberg
>>>>
>>>>
>>
>
>

I think the problem you are seeing is that you are launching the shell
script which, of course, has its own pid which is not the pid of the java
process it starts. Change your exec command to directly exec the java
command the shell script is executing, and I think this may work (echo the
command to get all the right paths and stuff).

You can also just do a soft kill, the broker should gracefully shut itself
down in response to this.

-Jay
On Sat, Jul 7, 2012 at 11:35 PM, Guy Doulberg <[EMAIL PROTECTED]>wrote:

> Hi Jay,
> Thanks for your replay.
>
> Not exactly,
>
> I can see that the pid the runit save, is deferent from the pid that I
> acctually have running on the machine , could it be that the launcher of
> the kafka broker forks another process with a different PID?
>
> Thanks.
>
> The only way I could kill the process is by using kill -9 , I suspect that
> runit doesn't use the
>
> On 07/05/2012 10:19 PM, Jay Kreps wrote:
>
>> I think what you are saying is that you don't have the process id for the
>> broker so you don't have the right thing to kill?
>>
>> Two solutions I know:
>> 1. Change your script to store the process id somewhere, and use that for
>> the kill
>> 2. Use pkill and just kill all kafka instances (only works if there is
>> only
>> one broker per node)
>>
>> -Jay
>>
>> On Thu, Jul 5, 2012 at 2:24 AM, Guy Doulberg <[EMAIL PROTECTED]>**
>> wrote:
>>
>>  Hi guys,
>>>
>>> In my deployment of Kafka I am using runit to control the service of the
>>> kafka broker.
>>>
>>> In the run script I have this line:
>>> #!/bin/sh
>>> exec 2>&1
>>>
>>> exec /opt/kafka/bin/kafka-server-****start.sh
>>> /etc/kafka/server.properties
>>>
>>>
>>>
>>>
>>> It works fine, if I want to start the broker, but if I want to stop or
>>> restart the process, the kafka broker doesn't get kill...
>>>
>>> So eventually the old kafka broker prevents from the new broker to start
>>> (Because the ports are already binded).
>>>
>>>
>>> Of course the running the same line of code in the shell and then ctrl-c
>>> ing works good (as you know kafka broker is not a daemon)
>>>
>>> I was wondering if someone has encountered this phenomena and solved it.
>>>
>>> Thanks,
>>>
>>> Guy Doulberg
>>>
>>>
>
>