Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> count of bytes stored in kafka


Copy link to this message
-
Re: count of bytes stored in kafka
Currently, there is no direct way of measuring this in 0.8 either. Do you
want to file a jira so that we can track this?

Thanks,

Jun
On Fri, Dec 27, 2013 at 9:27 PM, Barto Rael <[EMAIL PROTECTED]> wrote:

> Hi Otis,
>
> Thanks for response!
>
> We are using kafka 0.8.0. I was looking for some jmx metrix that can give
> count of messages/bytes stored in kafka.
>
> If we look into data stores like hbase we can see  how many messages are
> stored in hbase or if we look into search engine like elasticsearch then
> also we can see  how many messages are stored/indexed in elasticsearch. In
> similar way i was expecting that there should be direct way to see count
> of  messages or bytes stored in kafka without using any external tool.
>
> Is there any direct way to see count of messages or bytes stored in kafka ?
>
>
>
>
> On Sat, Dec 28, 2013 at 3:32 AM, Otis Gospodnetic <
> [EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > Not a direct way to see this, but we use Kafka 0.7.2 and monitor the disk
> > space on it, so that's one way to see *roughly* what is happening.
> > If you use something like SPM to monitor your Kafka nodes (see
> >
> >
> http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/
> > )
> > and if broker stores data on the disk partition that nothing else
> > writes
> > to, then you can look at disk usage and read/write rates for that
> specific
> > partition and thus get the more precise numbers.
> >
> > If there's a more direct way to see/get this number, I'm all eyeballs!
> >
> > Otis
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Thu, Dec 26, 2013 at 11:37 PM, Barto Rael <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi All,
> > >
> > > Can we see count of messages or bytes of a topic present in kafka
> > > using jconsole or any other way ?
> > >
> > >
> > > Jmx metric "BytesInperSec" doesn't show data stored in kafka.
> > > As per my understanding:
> > > "BytesInperSec" is count of bytes in a topic that have been received
> > > by the broker since the broker process started. It is just counter
> > > value that get incremented each time a new byte is received by broker.
> > > The value is reset to 0 when broker is restarted.
> > > Also the Byte count will not be decremented if any bytes are deleted
> > > from broker.
> > >
> > > Is there any way to check count of bytes stored in kafka even after
> > > kafka is restarted?
> > >
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB