Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> count of bytes stored in kafka


Copy link to this message
-
Re: count of bytes stored in kafka
Currently, there is no direct way of measuring this in 0.8 either. Do you
want to file a jira so that we can track this?

Thanks,

Jun
On Fri, Dec 27, 2013 at 9:27 PM, Barto Rael <[EMAIL PROTECTED]> wrote:

> Hi Otis,
>
> Thanks for response!
>
> We are using kafka 0.8.0. I was looking for some jmx metrix that can give
> count of messages/bytes stored in kafka.
>
> If we look into data stores like hbase we can see  how many messages are
> stored in hbase or if we look into search engine like elasticsearch then
> also we can see  how many messages are stored/indexed in elasticsearch. In
> similar way i was expecting that there should be direct way to see count
> of  messages or bytes stored in kafka without using any external tool.
>
> Is there any direct way to see count of messages or bytes stored in kafka ?
>
>
>
>
> On Sat, Dec 28, 2013 at 3:32 AM, Otis Gospodnetic <
> [EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > Not a direct way to see this, but we use Kafka 0.7.2 and monitor the disk
> > space on it, so that's one way to see *roughly* what is happening.
> > If you use something like SPM to monitor your Kafka nodes (see
> >
> >
> http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/
> > )
> > and if broker stores data on the disk partition that nothing else
> > writes
> > to, then you can look at disk usage and read/write rates for that
> specific
> > partition and thus get the more precise numbers.
> >
> > If there's a more direct way to see/get this number, I'm all eyeballs!
> >
> > Otis
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Thu, Dec 26, 2013 at 11:37 PM, Barto Rael <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi All,
> > >
> > > Can we see count of messages or bytes of a topic present in kafka
> > > using jconsole or any other way ?
> > >
> > >
> > > Jmx metric "BytesInperSec" doesn't show data stored in kafka.
> > > As per my understanding:
> > > "BytesInperSec" is count of bytes in a topic that have been received
> > > by the broker since the broker process started. It is just counter
> > > value that get incremented each time a new byte is received by broker.
> > > The value is reset to 0 when broker is restarted.
> > > Also the Byte count will not be decremented if any bytes are deleted
> > > from broker.
> > >
> > > Is there any way to check count of bytes stored in kafka even after
> > > kafka is restarted?
> > >
> >
>