Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Format of Kafka storage on disk

Copy link to this message
Re: Format of Kafka storage on disk
The DumpLogSegments should do that for you

bin/kafka-run-class.sh kafka.tools.DumpLogSegments

Option                                  Description

------                                  -----------

--deep-iteration                        if set, uses deep instead of

--files <file1, file2, ...>             REQUIRED: The comma separated list
                                          data and index log files to be
--max-message-size <Integer: size>      Size of largest message. (default:


--print-data-log                        if set, printing the messages
                                          when dumping data logs

--verify-index-only                     if set, just verify the index log

                                          without printing its content

or use the code as entry point for whatever you want to-do :)
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
On Fri, Jan 3, 2014 at 5:10 PM, Subbu Srinivasan <[EMAIL PROTECTED]>wrote:

> Is there any place where I can know about the internal structure of
> the log file where kafka stores the data. A topic has a .index and a .log
> file.
> I want to read the entire log file and parse the contents out.
> Thanks
> Subbu