Hi

I am using kafka-0.7.1. Is there a way i can make the broker publish some
metrics periodically in the log files. Something like "Topic:abc,
MsgsConsumed=N, DataSpeed=n msg/sec, ..... "

--
Regards
Rakesh Nair

Maybe you can use something like jmxtrans, or Jolokia (JMX over http)?

-Evan
Carry your candle, run to the darkness
Seek out the helpless, deceived and poor
Hold out your candle for all to see it
Take your candle, and go light your world
 
On Sep 30, 2012, at 8:38 PM, Rakesh Nair <[EMAIL PROTECTED]> wrote:

> Hi
>
> I am using kafka-0.7.1. Is there a way i can make the broker publish some
> metrics periodically in the log files. Something like "Topic:abc,
> MsgsConsumed=N, DataSpeed=n msg/sec, ..... "
>
> --
> Regards
> Rakesh Nair

We use JMXTrans, it's pretty easy to set up. Pro-tip: use the yml-to-config
generator included as a contrib script, it's much more concise.

On Sun, Sep 30, 2012 at 11:03 PM, Evan chan <[EMAIL PROTECTED]> wrote:

> Maybe you can use something like jmxtrans, or Jolokia (JMX over http)?
>
> -Evan
> Carry your candle, run to the darkness
> Seek out the helpless, deceived and poor
> Hold out your candle for all to see it
> Take your candle, and go light your world
>
>
>
> On Sep 30, 2012, at 8:38 PM, Rakesh Nair <[EMAIL PROTECTED]> wrote:
>
> > Hi
> >
> > I am using kafka-0.7.1. Is there a way i can make the broker publish some
> > metrics periodically in the log files. Something like "Topic:abc,
> > MsgsConsumed=N, DataSpeed=n msg/sec, ..... "
> >
> > --
> > Regards
> > Rakesh Nair
>

--
Matthew Rathbone
Foursquare | Software Engineer | Server Engineering Team
[EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
4sq<http://foursquare.com/rathboma>

Take a look at: http://metrics.codahale.com/
Used in production at Yammer and LinkedIn.

-John

On Mon, Oct 1, 2012 at 8:08 AM, Matthew Rathbone <[EMAIL PROTECTED]>wrote:

> We use JMXTrans, it's pretty easy to set up. Pro-tip: use the yml-to-config
> generator included as a contrib script, it's much more concise.
>
> On Sun, Sep 30, 2012 at 11:03 PM, Evan chan <[EMAIL PROTECTED]> wrote:
>
> > Maybe you can use something like jmxtrans, or Jolokia (JMX over http)?
> >
> > -Evan
> > Carry your candle, run to the darkness
> > Seek out the helpless, deceived and poor
> > Hold out your candle for all to see it
> > Take your candle, and go light your world
> >
> >
> >
> > On Sep 30, 2012, at 8:38 PM, Rakesh Nair <[EMAIL PROTECTED]> wrote:
> >
> > > Hi
> > >
> > > I am using kafka-0.7.1. Is there a way i can make the broker publish
> some
> > > metrics periodically in the log files. Something like "Topic:abc,
> > > MsgsConsumed=N, DataSpeed=n msg/sec, ..... "
> > >
> > > --
> > > Regards
> > > Rakesh Nair
> >
>
>
>
> --
> Matthew Rathbone
> Foursquare | Software Engineer | Server Engineering Team
> [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
> 4sq<http://foursquare.com/rathboma>
>

Any plans to incorporate coda hale metrics into Kafka?  That would be
awesome.

On Mon, Oct 1, 2012 at 11:26 AM, John Wang <[EMAIL PROTECTED]> wrote:

> Take a look at: http://metrics.codahale.com/
> Used in production at Yammer and LinkedIn.
>
> -John
>
> On Mon, Oct 1, 2012 at 8:08 AM, Matthew Rathbone <[EMAIL PROTECTED]
> >wrote:
>
> > We use JMXTrans, it's pretty easy to set up. Pro-tip: use the
> yml-to-config
> > generator included as a contrib script, it's much more concise.
> >
> > On Sun, Sep 30, 2012 at 11:03 PM, Evan chan <[EMAIL PROTECTED]> wrote:
> >
> > > Maybe you can use something like jmxtrans, or Jolokia (JMX over http)?
> > >
> > > -Evan
> > > Carry your candle, run to the darkness
> > > Seek out the helpless, deceived and poor
> > > Hold out your candle for all to see it
> > > Take your candle, and go light your world
> > >
> > >
> > >
> > > On Sep 30, 2012, at 8:38 PM, Rakesh Nair <[EMAIL PROTECTED]> wrote:
> > >
> > > > Hi
> > > >
> > > > I am using kafka-0.7.1. Is there a way i can make the broker publish
> > some
> > > > metrics periodically in the log files. Something like "Topic:abc,
> > > > MsgsConsumed=N, DataSpeed=n msg/sec, ..... "
> > > >
> > > > --
> > > > Regards
> > > > Rakesh Nair
> > >
> >
> >
> >
> > --
> > Matthew Rathbone
> > Foursquare | Software Engineer | Server Engineering Team
> > [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
> > 4sq<http://foursquare.com/rathboma>
> >
>

--
--
*Evan Chan*
Senior Software Engineer |
[EMAIL PROTECTED] | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>

For the metrics requested earlier in the thread, you can easily export
those metrics in the application layer.

-John

On Mon, Oct 1, 2012 at 11:34 AM, Evan Chan <[EMAIL PROTECTED]> wrote:

> Any plans to incorporate coda hale metrics into Kafka?  That would be
> awesome.
>
> On Mon, Oct 1, 2012 at 11:26 AM, John Wang <[EMAIL PROTECTED]> wrote:
>
> > Take a look at: http://metrics.codahale.com/
> > Used in production at Yammer and LinkedIn.
> >
> > -John
> >
> > On Mon, Oct 1, 2012 at 8:08 AM, Matthew Rathbone <[EMAIL PROTECTED]
> > >wrote:
> >
> > > We use JMXTrans, it's pretty easy to set up. Pro-tip: use the
> > yml-to-config
> > > generator included as a contrib script, it's much more concise.
> > >
> > > On Sun, Sep 30, 2012 at 11:03 PM, Evan chan <[EMAIL PROTECTED]> wrote:
> > >
> > > > Maybe you can use something like jmxtrans, or Jolokia (JMX over
> http)?
> > > >
> > > > -Evan
> > > > Carry your candle, run to the darkness
> > > > Seek out the helpless, deceived and poor
> > > > Hold out your candle for all to see it
> > > > Take your candle, and go light your world
> > > >
> > > >
> > > >
> > > > On Sep 30, 2012, at 8:38 PM, Rakesh Nair <[EMAIL PROTECTED]>
> wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > I am using kafka-0.7.1. Is there a way i can make the broker
> publish
> > > some
> > > > > metrics periodically in the log files. Something like "Topic:abc,
> > > > > MsgsConsumed=N, DataSpeed=n msg/sec, ..... "
> > > > >
> > > > > --
> > > > > Regards
> > > > > Rakesh Nair
> > > >
> > >
> > >
> > >
> > > --
> > > Matthew Rathbone
> > > Foursquare | Software Engineer | Server Engineering Team
> > > [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
> > > 4sq<http://foursquare.com/rathboma>
> > >
> >
>
>
>
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> [EMAIL PROTECTED] | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>
>

I too would like to see codahale metrics in to the broker.

We did implement it on our producers and consumers but it would be nice to
have the over all view of the world from the broker.

For now we are using JMXTrans as well, seems to work ok.

On Mon, Oct 1, 2012 at 11:38 AM, John Wang <[EMAIL PROTECTED]> wrote:

> For the metrics requested earlier in the thread, you can easily export
> those metrics in the application layer.
>
> -John
>
> On Mon, Oct 1, 2012 at 11:34 AM, Evan Chan <[EMAIL PROTECTED]> wrote:
>
> > Any plans to incorporate coda hale metrics into Kafka?  That would be
> > awesome.
> >
> > On Mon, Oct 1, 2012 at 11:26 AM, John Wang <[EMAIL PROTECTED]> wrote:
> >
> > > Take a look at: http://metrics.codahale.com/
> > > Used in production at Yammer and LinkedIn.
> > >
> > > -John
> > >
> > > On Mon, Oct 1, 2012 at 8:08 AM, Matthew Rathbone <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > We use JMXTrans, it's pretty easy to set up. Pro-tip: use the
> > > yml-to-config
> > > > generator included as a contrib script, it's much more concise.
> > > >
> > > > On Sun, Sep 30, 2012 at 11:03 PM, Evan chan <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Maybe you can use something like jmxtrans, or Jolokia (JMX over
> > http)?
> > > > >
> > > > > -Evan
> > > > > Carry your candle, run to the darkness
> > > > > Seek out the helpless, deceived and poor
> > > > > Hold out your candle for all to see it
> > > > > Take your candle, and go light your world
> > > > >
> > > > >
> > > > >
> > > > > On Sep 30, 2012, at 8:38 PM, Rakesh Nair <[EMAIL PROTECTED]>
> > wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > I am using kafka-0.7.1. Is there a way i can make the broker
> > publish
> > > > some
> > > > > > metrics periodically in the log files. Something like "Topic:abc,
> > > > > > MsgsConsumed=N, DataSpeed=n msg/sec, ..... "
> > > > > >
> > > > > > --
> > > > > > Regards
> > > > > > Rakesh Nair
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Matthew Rathbone
> > > > Foursquare | Software Engineer | Server Engineering Team
> > > > [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
> > > > 4sq<http://foursquare.com/rathboma>
> > > >
> > >
> >
> >
> >
> > --
> > --
> > *Evan Chan*
> > Senior Software Engineer |
> > [EMAIL PROTECTED] | (650) 996-4600
> > www.ooyala.com | blog <http://www.ooyala.com/blog> |
> > @ooyala<http://www.twitter.com/ooyala>
> >
>

We started using coda hale metrics in 0.8. Some portions of that could be
ported into trunk/0.7 if anyone is interested in contributing a patch.

Thanks,

Joel

On Mon, Oct 1, 2012 at 12:04 PM, Matt Christiansen <[EMAIL PROTECTED]> wrote:

> I too would like to see codahale metrics in to the broker.
>
> We did implement it on our producers and consumers but it would be nice to
> have the over all view of the world from the broker.
>
> For now we are using JMXTrans as well, seems to work ok.
>
> On Mon, Oct 1, 2012 at 11:38 AM, John Wang <[EMAIL PROTECTED]> wrote:
>
> > For the metrics requested earlier in the thread, you can easily export
> > those metrics in the application layer.
> >
> > -John
> >
> > On Mon, Oct 1, 2012 at 11:34 AM, Evan Chan <[EMAIL PROTECTED]> wrote:
> >
> > > Any plans to incorporate coda hale metrics into Kafka?  That would be
> > > awesome.
> > >
> > > On Mon, Oct 1, 2012 at 11:26 AM, John Wang <[EMAIL PROTECTED]>
> wrote:
> > >
> > > > Take a look at: http://metrics.codahale.com/
> > > > Used in production at Yammer and LinkedIn.
> > > >
> > > > -John
> > > >
> > > > On Mon, Oct 1, 2012 at 8:08 AM, Matthew Rathbone <
> > [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > We use JMXTrans, it's pretty easy to set up. Pro-tip: use the
> > > > yml-to-config
> > > > > generator included as a contrib script, it's much more concise.
> > > > >
> > > > > On Sun, Sep 30, 2012 at 11:03 PM, Evan chan <[EMAIL PROTECTED]> wrote:
> > > > >
> > > > > > Maybe you can use something like jmxtrans, or Jolokia (JMX over
> > > http)?
> > > > > >
> > > > > > -Evan
> > > > > > Carry your candle, run to the darkness
> > > > > > Seek out the helpless, deceived and poor
> > > > > > Hold out your candle for all to see it
> > > > > > Take your candle, and go light your world
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sep 30, 2012, at 8:38 PM, Rakesh Nair <[EMAIL PROTECTED]>
> > > wrote:
> > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > I am using kafka-0.7.1. Is there a way i can make the broker
> > > publish
> > > > > some
> > > > > > > metrics periodically in the log files. Something like
> "Topic:abc,
> > > > > > > MsgsConsumed=N, DataSpeed=n msg/sec, ..... "
> > > > > > >
> > > > > > > --
> > > > > > > Regards
> > > > > > > Rakesh Nair
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Matthew Rathbone
> > > > > Foursquare | Software Engineer | Server Engineering Team
> > > > > [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
> > > > > 4sq<http://foursquare.com/rathboma>
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > --
> > > *Evan Chan*
> > > Senior Software Engineer |
> > > [EMAIL PROTECTED] | (650) 996-4600
> > > www.ooyala.com | blog <http://www.ooyala.com/blog> |
> > > @ooyala<http://www.twitter.com/ooyala>
> > >
> >
>

Well the point is that JMX doesn't work in EC2, so having coda hale metrics from Kafka directly would be a win.

-Evan
Carry your candle, run to the darkness
Seek out the helpless, deceived and poor
Hold out your candle for all to see it
Take your candle, and go light your world
 
On Oct 1, 2012, at 11:38 AM, John Wang <[EMAIL PROTECTED]> wrote:

> For the metrics requested earlier in the thread, you can easily export
> those metrics in the application layer.
>
> -John
>
> On Mon, Oct 1, 2012 at 11:34 AM, Evan Chan <[EMAIL PROTECTED]> wrote:
>
>> Any plans to incorporate coda hale metrics into Kafka?  That would be
>> awesome.
>>
>> On Mon, Oct 1, 2012 at 11:26 AM, John Wang <[EMAIL PROTECTED]> wrote:
>>
>>> Take a look at: http://metrics.codahale.com/
>>> Used in production at Yammer and LinkedIn.
>>>
>>> -John
>>>
>>> On Mon, Oct 1, 2012 at 8:08 AM, Matthew Rathbone <[EMAIL PROTECTED]
>>>> wrote:
>>>
>>>> We use JMXTrans, it's pretty easy to set up. Pro-tip: use the
>>> yml-to-config
>>>> generator included as a contrib script, it's much more concise.
>>>>
>>>> On Sun, Sep 30, 2012 at 11:03 PM, Evan chan <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Maybe you can use something like jmxtrans, or Jolokia (JMX over
>> http)?
>>>>>
>>>>> -Evan
>>>>> Carry your candle, run to the darkness
>>>>> Seek out the helpless, deceived and poor
>>>>> Hold out your candle for all to see it
>>>>> Take your candle, and go light your world
>>>>>
>>>>>
>>>>>
>>>>> On Sep 30, 2012, at 8:38 PM, Rakesh Nair <[EMAIL PROTECTED]>
>> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I am using kafka-0.7.1. Is there a way i can make the broker
>> publish
>>>> some
>>>>>> metrics periodically in the log files. Something like "Topic:abc,
>>>>>> MsgsConsumed=N, DataSpeed=n msg/sec, ..... "
>>>>>>
>>>>>> --
>>>>>> Regards
>>>>>> Rakesh Nair
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Matthew Rathbone
>>>> Foursquare | Software Engineer | Server Engineering Team
>>>> [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
>>>> 4sq<http://foursquare.com/rathboma>
>>>>
>>>
>>
>>
>>
>> --
>> --
>> *Evan Chan*
>> Senior Software Engineer |
>> [EMAIL PROTECTED] | (650) 996-4600
>> www.ooyala.com | blog <http://www.ooyala.com/blog> |
>> @ooyala<http://www.twitter.com/ooyala>
>>

Hey Evan,

Not sure what you mean there, we're using JMX in EC2, what issues have you
seen? You should just be able to open a port and point jmxtrans at it.

On Mon, Oct 1, 2012 at 12:08 PM, Evan chan <[EMAIL PROTECTED]> wrote:

> Well the point is that JMX doesn't work in EC2, so having coda hale
> metrics from Kafka directly would be a win.
>
> -Evan
> Carry your candle, run to the darkness
> Seek out the helpless, deceived and poor
> Hold out your candle for all to see it
> Take your candle, and go light your world
>
>
>
> On Oct 1, 2012, at 11:38 AM, John Wang <[EMAIL PROTECTED]> wrote:
>
> > For the metrics requested earlier in the thread, you can easily export
> > those metrics in the application layer.
> >
> > -John
> >
> > On Mon, Oct 1, 2012 at 11:34 AM, Evan Chan <[EMAIL PROTECTED]> wrote:
> >
> >> Any plans to incorporate coda hale metrics into Kafka?  That would be
> >> awesome.
> >>
> >> On Mon, Oct 1, 2012 at 11:26 AM, John Wang <[EMAIL PROTECTED]> wrote:
> >>
> >>> Take a look at: http://metrics.codahale.com/
> >>> Used in production at Yammer and LinkedIn.
> >>>
> >>> -John
> >>>
> >>> On Mon, Oct 1, 2012 at 8:08 AM, Matthew Rathbone <
> [EMAIL PROTECTED]
> >>>> wrote:
> >>>
> >>>> We use JMXTrans, it's pretty easy to set up. Pro-tip: use the
> >>> yml-to-config
> >>>> generator included as a contrib script, it's much more concise.
> >>>>
> >>>> On Sun, Sep 30, 2012 at 11:03 PM, Evan chan <[EMAIL PROTECTED]> wrote:
> >>>>
> >>>>> Maybe you can use something like jmxtrans, or Jolokia (JMX over
> >> http)?
> >>>>>
> >>>>> -Evan
> >>>>> Carry your candle, run to the darkness
> >>>>> Seek out the helpless, deceived and poor
> >>>>> Hold out your candle for all to see it
> >>>>> Take your candle, and go light your world
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Sep 30, 2012, at 8:38 PM, Rakesh Nair <[EMAIL PROTECTED]>
> >> wrote:
> >>>>>
> >>>>>> Hi
> >>>>>>
> >>>>>> I am using kafka-0.7.1. Is there a way i can make the broker
> >> publish
> >>>> some
> >>>>>> metrics periodically in the log files. Something like "Topic:abc,
> >>>>>> MsgsConsumed=N, DataSpeed=n msg/sec, ..... "
> >>>>>>
> >>>>>> --
> >>>>>> Regards
> >>>>>> Rakesh Nair
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Matthew Rathbone
> >>>> Foursquare | Software Engineer | Server Engineering Team
> >>>> [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
> >>>> 4sq<http://foursquare.com/rathboma>
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> --
> >> *Evan Chan*
> >> Senior Software Engineer |
> >> [EMAIL PROTECTED] | (650) 996-4600
> >> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> >> @ooyala<http://www.twitter.com/ooyala>
> >>
>

--
Matthew Rathbone
Foursquare | Software Engineer | Server Engineering Team
[EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
4sq<http://foursquare.com/rathboma>

What I mean is that we can't use JConsole, etc.  from outside of EC2 into
the EC2 machine.  I'm aware one can use jmxtrans on the same box.

On Mon, Oct 1, 2012 at 12:50 PM, Matthew Rathbone <[EMAIL PROTECTED]>wrote:

> Hey Evan,
>
> Not sure what you mean there, we're using JMX in EC2, what issues have you
> seen? You should just be able to open a port and point jmxtrans at it.
>
> On Mon, Oct 1, 2012 at 12:08 PM, Evan chan <[EMAIL PROTECTED]> wrote:
>
> > Well the point is that JMX doesn't work in EC2, so having coda hale
> > metrics from Kafka directly would be a win.
> >
> > -Evan
> > Carry your candle, run to the darkness
> > Seek out the helpless, deceived and poor
> > Hold out your candle for all to see it
> > Take your candle, and go light your world
> >
> >
> >
> > On Oct 1, 2012, at 11:38 AM, John Wang <[EMAIL PROTECTED]> wrote:
> >
> > > For the metrics requested earlier in the thread, you can easily export
> > > those metrics in the application layer.
> > >
> > > -John
> > >
> > > On Mon, Oct 1, 2012 at 11:34 AM, Evan Chan <[EMAIL PROTECTED]> wrote:
> > >
> > >> Any plans to incorporate coda hale metrics into Kafka?  That would be
> > >> awesome.
> > >>
> > >> On Mon, Oct 1, 2012 at 11:26 AM, John Wang <[EMAIL PROTECTED]>
> wrote:
> > >>
> > >>> Take a look at: http://metrics.codahale.com/
> > >>> Used in production at Yammer and LinkedIn.
> > >>>
> > >>> -John
> > >>>
> > >>> On Mon, Oct 1, 2012 at 8:08 AM, Matthew Rathbone <
> > [EMAIL PROTECTED]
> > >>>> wrote:
> > >>>
> > >>>> We use JMXTrans, it's pretty easy to set up. Pro-tip: use the
> > >>> yml-to-config
> > >>>> generator included as a contrib script, it's much more concise.
> > >>>>
> > >>>> On Sun, Sep 30, 2012 at 11:03 PM, Evan chan <[EMAIL PROTECTED]> wrote:
> > >>>>
> > >>>>> Maybe you can use something like jmxtrans, or Jolokia (JMX over
> > >> http)?
> > >>>>>
> > >>>>> -Evan
> > >>>>> Carry your candle, run to the darkness
> > >>>>> Seek out the helpless, deceived and poor
> > >>>>> Hold out your candle for all to see it
> > >>>>> Take your candle, and go light your world
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Sep 30, 2012, at 8:38 PM, Rakesh Nair <[EMAIL PROTECTED]>
> > >> wrote:
> > >>>>>
> > >>>>>> Hi
> > >>>>>>
> > >>>>>> I am using kafka-0.7.1. Is there a way i can make the broker
> > >> publish
> > >>>> some
> > >>>>>> metrics periodically in the log files. Something like "Topic:abc,
> > >>>>>> MsgsConsumed=N, DataSpeed=n msg/sec, ..... "
> > >>>>>>
> > >>>>>> --
> > >>>>>> Regards
> > >>>>>> Rakesh Nair
> > >>>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Matthew Rathbone
> > >>>> Foursquare | Software Engineer | Server Engineering Team
> > >>>> [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
> > >>>> 4sq<http://foursquare.com/rathboma>
> > >>>>
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> --
> > >> *Evan Chan*
> > >> Senior Software Engineer |
> > >> [EMAIL PROTECTED] | (650) 996-4600
> > >> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> > >> @ooyala<http://www.twitter.com/ooyala>
> > >>
> >
>
>
>
> --
> Matthew Rathbone
> Foursquare | Software Engineer | Server Engineering Team
> [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
> 4sq<http://foursquare.com/rathboma>
>

--
--
*Evan Chan*
Senior Software Engineer |
[EMAIL PROTECTED] | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>

I'm using JConsole from outside EC2 to connect to our EC2 boxes. I'll be
happy to share the script if you like.

Pato

On Tue, Oct 2, 2012 at 10:02 AM, Evan Chan <[EMAIL PROTECTED]> wrote:

> What I mean is that we can't use JConsole, etc.  from outside of EC2 into
> the EC2 machine.  I'm aware one can use jmxtrans on the same box.
>
> On Mon, Oct 1, 2012 at 12:50 PM, Matthew Rathbone <[EMAIL PROTECTED]
> >wrote:
>
> > Hey Evan,
> >
> > Not sure what you mean there, we're using JMX in EC2, what issues have
> you
> > seen? You should just be able to open a port and point jmxtrans at it.
> >
> > On Mon, Oct 1, 2012 at 12:08 PM, Evan chan <[EMAIL PROTECTED]> wrote:
> >
> > > Well the point is that JMX doesn't work in EC2, so having coda hale
> > > metrics from Kafka directly would be a win.
> > >
> > > -Evan
> > > Carry your candle, run to the darkness
> > > Seek out the helpless, deceived and poor
> > > Hold out your candle for all to see it
> > > Take your candle, and go light your world
> > >
> > >
> > >
> > > On Oct 1, 2012, at 11:38 AM, John Wang <[EMAIL PROTECTED]> wrote:
> > >
> > > > For the metrics requested earlier in the thread, you can easily
> export
> > > > those metrics in the application layer.
> > > >
> > > > -John
> > > >
> > > > On Mon, Oct 1, 2012 at 11:34 AM, Evan Chan <[EMAIL PROTECTED]> wrote:
> > > >
> > > >> Any plans to incorporate coda hale metrics into Kafka?  That would
> be
> > > >> awesome.
> > > >>
> > > >> On Mon, Oct 1, 2012 at 11:26 AM, John Wang <[EMAIL PROTECTED]>
> > wrote:
> > > >>
> > > >>> Take a look at: http://metrics.codahale.com/
> > > >>> Used in production at Yammer and LinkedIn.
> > > >>>
> > > >>> -John
> > > >>>
> > > >>> On Mon, Oct 1, 2012 at 8:08 AM, Matthew Rathbone <
> > > [EMAIL PROTECTED]
> > > >>>> wrote:
> > > >>>
> > > >>>> We use JMXTrans, it's pretty easy to set up. Pro-tip: use the
> > > >>> yml-to-config
> > > >>>> generator included as a contrib script, it's much more concise.
> > > >>>>
> > > >>>> On Sun, Sep 30, 2012 at 11:03 PM, Evan chan <[EMAIL PROTECTED]>
> wrote:
> > > >>>>
> > > >>>>> Maybe you can use something like jmxtrans, or Jolokia (JMX over
> > > >> http)?
> > > >>>>>
> > > >>>>> -Evan
> > > >>>>> Carry your candle, run to the darkness
> > > >>>>> Seek out the helpless, deceived and poor
> > > >>>>> Hold out your candle for all to see it
> > > >>>>> Take your candle, and go light your world
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> On Sep 30, 2012, at 8:38 PM, Rakesh Nair <[EMAIL PROTECTED]>
> > > >> wrote:
> > > >>>>>
> > > >>>>>> Hi
> > > >>>>>>
> > > >>>>>> I am using kafka-0.7.1. Is there a way i can make the broker
> > > >> publish
> > > >>>> some
> > > >>>>>> metrics periodically in the log files. Something like
> "Topic:abc,
> > > >>>>>> MsgsConsumed=N, DataSpeed=n msg/sec, ..... "
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>> Regards
> > > >>>>>> Rakesh Nair
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Matthew Rathbone
> > > >>>> Foursquare | Software Engineer | Server Engineering Team
> > > >>>> [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma>
> |
> > > >>>> 4sq<http://foursquare.com/rathboma>
> > > >>>>
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> --
> > > >> *Evan Chan*
> > > >> Senior Software Engineer |
> > > >> [EMAIL PROTECTED] | (650) 996-4600
> > > >> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> > > >> @ooyala<http://www.twitter.com/ooyala>
> > > >>
> > >
> >
> >
> >
> > --
> > Matthew Rathbone
> > Foursquare | Software Engineer | Server Engineering Team
> > [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
> > 4sq<http://foursquare.com/rathboma>
> >
>
>
>
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> [EMAIL PROTECTED] | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>
>

Ah, yeah we manage to do that using a VPN.

On Tue, Oct 2, 2012 at 12:30 PM, Patricio Echagüe <[EMAIL PROTECTED]>wrote:

> I'm using JConsole from outside EC2 to connect to our EC2 boxes. I'll be
> happy to share the script if you like.
>
> Pato
>
> On Tue, Oct 2, 2012 at 10:02 AM, Evan Chan <[EMAIL PROTECTED]> wrote:
>
> > What I mean is that we can't use JConsole, etc.  from outside of EC2 into
> > the EC2 machine.  I'm aware one can use jmxtrans on the same box.
> >
> > On Mon, Oct 1, 2012 at 12:50 PM, Matthew Rathbone <
> [EMAIL PROTECTED]
> > >wrote:
> >
> > > Hey Evan,
> > >
> > > Not sure what you mean there, we're using JMX in EC2, what issues have
> > you
> > > seen? You should just be able to open a port and point jmxtrans at it.
> > >
> > > On Mon, Oct 1, 2012 at 12:08 PM, Evan chan <[EMAIL PROTECTED]> wrote:
> > >
> > > > Well the point is that JMX doesn't work in EC2, so having coda hale
> > > > metrics from Kafka directly would be a win.
> > > >
> > > > -Evan
> > > > Carry your candle, run to the darkness
> > > > Seek out the helpless, deceived and poor
> > > > Hold out your candle for all to see it
> > > > Take your candle, and go light your world
> > > >
> > > >
> > > >
> > > > On Oct 1, 2012, at 11:38 AM, John Wang <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > For the metrics requested earlier in the thread, you can easily
> > export
> > > > > those metrics in the application layer.
> > > > >
> > > > > -John
> > > > >
> > > > > On Mon, Oct 1, 2012 at 11:34 AM, Evan Chan <[EMAIL PROTECTED]> wrote:
> > > > >
> > > > >> Any plans to incorporate coda hale metrics into Kafka?  That would
> > be
> > > > >> awesome.
> > > > >>
> > > > >> On Mon, Oct 1, 2012 at 11:26 AM, John Wang <[EMAIL PROTECTED]>
> > > wrote:
> > > > >>
> > > > >>> Take a look at: http://metrics.codahale.com/
> > > > >>> Used in production at Yammer and LinkedIn.
> > > > >>>
> > > > >>> -John
> > > > >>>
> > > > >>> On Mon, Oct 1, 2012 at 8:08 AM, Matthew Rathbone <
> > > > [EMAIL PROTECTED]
> > > > >>>> wrote:
> > > > >>>
> > > > >>>> We use JMXTrans, it's pretty easy to set up. Pro-tip: use the
> > > > >>> yml-to-config
> > > > >>>> generator included as a contrib script, it's much more concise.
> > > > >>>>
> > > > >>>> On Sun, Sep 30, 2012 at 11:03 PM, Evan chan <[EMAIL PROTECTED]>
> > wrote:
> > > > >>>>
> > > > >>>>> Maybe you can use something like jmxtrans, or Jolokia (JMX over
> > > > >> http)?
> > > > >>>>>
> > > > >>>>> -Evan
> > > > >>>>> Carry your candle, run to the darkness
> > > > >>>>> Seek out the helpless, deceived and poor
> > > > >>>>> Hold out your candle for all to see it
> > > > >>>>> Take your candle, and go light your world
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Sep 30, 2012, at 8:38 PM, Rakesh Nair <[EMAIL PROTECTED]>
> > > > >> wrote:
> > > > >>>>>
> > > > >>>>>> Hi
> > > > >>>>>>
> > > > >>>>>> I am using kafka-0.7.1. Is there a way i can make the broker
> > > > >> publish
> > > > >>>> some
> > > > >>>>>> metrics periodically in the log files. Something like
> > "Topic:abc,
> > > > >>>>>> MsgsConsumed=N, DataSpeed=n msg/sec, ..... "
> > > > >>>>>>
> > > > >>>>>> --
> > > > >>>>>> Regards
> > > > >>>>>> Rakesh Nair
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> Matthew Rathbone
> > > > >>>> Foursquare | Software Engineer | Server Engineering Team
> > > > >>>> [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma
> >
> > |
> > > > >>>> 4sq<http://foursquare.com/rathboma>
> > > > >>>>
> > > > >>>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> --
> > > > >> *Evan Chan*
> > > > >> Senior Software Engineer |
> > > > >> [EMAIL PROTECTED] | (650) 996-4600
> > > > >> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> > > > >> @ooyala<http://www.twitter.com/ooyala>
> > > > >>
> > > >
> > >
> > >
> > >
> > > --
> > > Matthew Rathbone
> > > Foursquare | Software Engineer | Server Engineering Team
> > > [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |

Matthew Rathbone
Foursquare | Software Engineer | Server Engineering Team
[EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
4sq<http://foursquare.com/rathboma>