Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Topic corruption from hardware failure (0.7.1)


Copy link to this message
-
Re: Topic corruption from hardware failure (0.7.1)
Jun Rao 2013-07-22, 05:30
For #1, one potential cause is that someone was consuming a wrong topic
name (e.g., datapoints-wr/ite-34). In 0.7, this will accidentally create a
directory datapoints-wr in the kafka log root dir. The problem has been
fixed in 0.8 by restricting the topic names to alpha-numeric plus "-" and
"_".

Thanks,

Jun
On Fri, Jul 19, 2013 at 7:45 AM, Blake Smith <[EMAIL PROTECTED]>wrote:

> Just to close the loop here for posterity:
>
> 1. For the directory topic name corruption, it looks like there's still an
> outstanding issue in JIRA: https://issues.apache.org/jira/browse/KAFKA-411
> 2. Ensuring log recovery is run seems to be fixed in
> commit 75fc5eab35aa33cffd9c09a2070dfe287db0ef4e (
> https://issues.apache.org/jira/browse/KAFKA-188)
>
> Thanks for your help Jay, it looks like we'll have to start the 0.7.1 ->
> 0.8 upgrade process sooner than I thought.
>
>
> On Wed, Jul 17, 2013 at 12:32 PM, Blake Smith <[EMAIL PROTECTED]
> >wrote:
>
> > 1. Cool. For now I'll write this off as a fluke side-effect of a unsafe
> > hardware shutdown. If I come up with a reproducible test case, I'll send
> it
> > (or a patch) your way.
> >
> > 2. After the process came up after the crash, recovery tried to run on
> > each partition, but because of the malformed directory issue it looks
> like
> > recovery never ran on partition 11 (the log with the corrupt message).
> >
> > 40472 [main] INFO  kafka.log.Log  - Loading the last segment
> > /var/kafka/datapoints-write-33/00000000093417473922.kafka in mutable
> mode,
> > recovery true
> > 44032 [main] INFO  kafka.message.FileMessageSet  - recover high water
> > mark:199897864
> > 44032 [main] INFO  kafka.message.FileMessageSet  - Recovery succeeded in
> 3
> > seconds. 0 bytes truncated.
> > 44033 [main] INFO  kafka.log.LogManager  - Loading log
> > 'datapoints-write-35'
> > 44044 [main] INFO  kafka.log.Log  - Loading the last segment
> > /var/kafka/datapoints-write-35/00000000092880891040.kafka in mutable
> mode,
> > recovery true
> > 48836 [main] INFO  kafka.message.FileMessageSet  - recover high water
> > mark:268123698
> > 48836 [main] INFO  kafka.message.FileMessageSet  - Recovery succeeded in
> 4
> > seconds. 0 bytes truncated.
> > 48859 [main] INFO  kafka.log.LogManager  - Loading log 'datapoints-wr'
> > 48911 [main] FATAL kafka.server.KafkaServerStartable  - Fatal error
> during
> > KafkaServerStable startup. Prepare to shutdown
> > java.lang.NumberFormatException: For input string: "wr"
> >         at
> >
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> >         at java.lang.Integer.parseInt(Integer.java:481)
> >         at java.lang.Integer.parseInt(Integer.java:514)
> >         at
> > scala.collection.immutable.StringLike$class.toInt(StringLike.scala:207)
> >         at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
> >         at kafka.utils.Utils$.getTopicPartition(Utils.scala:558)
> >         at kafka.log.LogManager$$anonfun$4.apply(LogManager.scala:71)
> >         at kafka.log.LogManager$$anonfun$4.apply(LogManager.scala:65)
> >         at
> >
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
> >         at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
> >         at kafka.log.LogManager.<init>(LogManager.scala:65)
> >         at kafka.server.KafkaServer.startup(KafkaServer.scala:58)
> >         at
> > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> >         at kafka.Kafka$.main(Kafka.scala:50)
> >         at kafka.Kafka.main(Kafka.scala)
> > 48912 [main] INFO  kafka.server.KafkaServer  - Shutting down Kafka server
> > 48913 [main] INFO  kafka.utils.KafkaScheduler  - shutdown scheduler
> > kafka-logcleaner-
> > 48913 [main] INFO  kafka.server.KafkaServer  - Kafka server shut down
> > completed
> > 0    [main] INFO  kafka.utils.Utils$  - The number of partitions for
> topic
> >  datapoints-write : 50
> > 2    [main] INFO  kafka.utils.Utils$  - The number of partitions for