Kafka, mail # user - Re: We have an invalid message in a segment, how to deal with it? - 2013-04-13, 18:32
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Re: We have an invalid message in a segment, how to deal with it?
I think the error you are seeing is due to the gap in the log. In 0.7
we validate that the log is contiguous and by deleting a segment you
are missing a chunk which would lead to problems in fetches for
offsets in that range. You can always safely delete a prefix of the
log (i.e. that segment and everything before it). You could also
rename the files to be contiguous if you had a lot of patience (i.e.
the nth file needs to have a name corresponding to the n-1st file
+length(n-1st file) if that makes sense...

I guess the forensics question is how we ended up rolling the log with
an invalid message, the broker should kill itself and then fix the log
on recovery when that occurs.

-Jay

On Sat, Apr 13, 2013 at 11:27 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB