Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - Does Kafka "message manager" tool (read messages from console, delete messages) exist?


Copy link to this message
-
Does Kafka "message manager" tool (read messages from console, delete messages) exist?
Vladimir Tretyakov 2013-07-23, 12:22
Hi, we use Kafka (0.7.2 version) in our product as message delivery service
and message storage (our retention time 48h) system. Thanks a lot for
perfect tool/lib. Now we really need 2 additional features:
1. Sometimes we need look at our messages we have in Kafka from command
line, for example:

~$ kafkaTool get topic=my_topic_name from=timestamp_1 to=timestamp_2 | grep
'Exception'

We need this feature because we want to have ability to see our "raw"
messages body.

2. Sometimes we have to delete messages older than given timestamp. Or in
other words we have to change offset for all consumers for given topic.
(But we don't like to work with offset because we know only timestamp we
like to move our consumers to). Example:

~$ kafkaTool purge topic=my_topic_name from=timestamp_1 to=timestamp_2

We need this feature because we want to skip processing of some messages by
our regular consumers.
Is these 2 features exist? If not, what is the best way to do this using
the existing APIs?

According to https://issues.apache.org/jira/browse/KAFKA-260 there is plan
to add timestamp to message. Is it possible, or is it planned, to have ...
something like FetchRequest, but with "timestamp" as parameter instead of
"offset"?

If this is currently not possible, should I open a JIRA issue?

Best regards, Vladimir.