Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> We have an invalid message in a segment, how to deal with it?

Matthew Rathbone 2013-04-13, 17:31
Copy link to this message
Re: We have an invalid message in a segment, how to deal with it?
You should be able to just bounce the broker. Our default policy is
that if we run out of space we shut down the broker automatically as
in that case there is no guarantee on what has been written to disk.
On startup if a clean shutdown hasn't been performed the broker should
run a recovery procedure on the log that includes checksumming all
messages. Invalid messages will be removed. Sounds like this isn't
what happened?


On Sat, Apr 13, 2013 at 10:30 AM, Matthew Rathbone
> Hey guys,
> Due to a disk filling up, one of the segments has an invalid message in it.
> I have verified this using DumpLogSegments.
> How do I deal with this now? the invalid message is causing our Hadoop
> Consumer to fail.
> Is there a way to remove the invalid message from the segment? Removing the
> whole segment causes the broker to fail on startup with an error:
> java.lang.IllegalStateException: The following segments don't validate:
> <bad-file>, <bad-file+1>
> I'm happy losing that file if needs be, but I need to get this broker back
> up asap.
> --
> Matthew Rathbone
> Foursquare | Software Engineer | Server Engineering Team
> [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
> 4sq<http://foursquare.com/rathboma>

Jay Kreps 2013-04-13, 18:32
Matthew Rathbone 2013-04-13, 18:51