Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Does Kafka "message manager" tool (read messages from console, delete messages) exist?


Copy link to this message
-
Re: Does Kafka "message manager" tool (read messages from console, delete messages) exist?
1. getOffsetBefore returns even coarser level of info. It only returns the
offset at the beginning of each log segment. There are some thoughts of
improving that. However, that's probably a post 0.8.1 item.

2. In 0.7, you need to bounce the broker to pick up the config change. In
0.8.1, restarting the broker will not be needed. In Kafka, you can only
delete data older than a timestamp.

Thanks,

Jun
On Tue, Jul 23, 2013 at 11:33 PM, Vladimir Tretyakov <
[EMAIL PROTECTED]> wrote:

> Hi Jun Rao and thanks for your answer.
> Re 1: Will try to do it way you have described. As I understood
> "getOffsetBefore"
> returns offset for set of compressed messages. In my case it will mean that
> my "from=timestamp_1" and "to=timestamp_2"  will work roughly (I will have
> to double check timestamp for each message inside tool and decide fit this
> message my criteria or not), right? Is there any chance to get API with
> "honest" offset for particular time?
>
> Re 2: Can I use https://issues.apache.org/jira/browse/KAFKA-554 for
> configuring "topic.log.roll.hours" on the fly? Should I restart Kafka after
> changes in config by
> KAFKA-554<https://issues.apache.org/jira/browse/KAFKA-554>?
> How about "to=timestamp_2". With "topic.log.roll.hours" I can define only
> one border for time range. What about second (right border on timeline)?
>
> Thanks, Vladimir.
>
>
>
> On Wed, Jul 24, 2013 at 7:42 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > For 1, you can write a tool that uses our getOffsetBefore api and
> > SimpleConsumer. The offset returned is at the log segment boundary. So
> it's
> > not going to match the specified time precisely.
> >
> > For 2, you can configure a topic to be rolled by timestamp
> > ("log.roll.hours" and "topic.log.roll.hours"). Then set the retention
> time
> > accordingly.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Jul 23, 2013 at 5:21 AM, Vladimir Tretyakov <
> > [EMAIL PROTECTED]> wrote:
> >
> > > Hi, we use Kafka (0.7.2 version) in our product as message delivery
> > service
> > > and message storage (our retention time 48h) system. Thanks a lot for
> > > perfect tool/lib. Now we really need 2 additional features:
> > > 1. Sometimes we need look at our messages we have in Kafka from command
> > > line, for example:
> > >
> > > ~$ kafkaTool get topic=my_topic_name from=timestamp_1 to=timestamp_2 |
> > grep
> > > 'Exception'
> > >
> > > We need this feature because we want to have ability to see our "raw"
> > > messages body.
> > >
> > > 2. Sometimes we have to delete messages older than given timestamp. Or
> in
> > > other words we have to change offset for all consumers for given topic.
> > > (But we don't like to work with offset because we know only timestamp
> we
> > > like to move our consumers to). Example:
> > >
> > > ~$ kafkaTool purge topic=my_topic_name from=timestamp_1 to=timestamp_2
> > >
> > > We need this feature because we want to skip processing of some
> messages
> > by
> > > our regular consumers.
> > >
> > >
> > > Is these 2 features exist? If not, what is the best way to do this
> using
> > > the existing APIs?
> > >
> > > According to https://issues.apache.org/jira/browse/KAFKA-260 there is
> > plan
> > > to add timestamp to message. Is it possible, or is it planned, to have
> > ...
> > > something like FetchRequest, but with "timestamp" as parameter instead
> of
> > > "offset"?
> > >
> > > If this is currently not possible, should I open a JIRA issue?
> > >
> > > Best regards, Vladimir.
> > >
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB