Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Topic corruption from hardware failure (0.7.1)


Copy link to this message
-
Re: Topic corruption from hardware failure (0.7.1)
For #1, one potential cause is that someone was consuming a wrong topic
name (e.g., datapoints-wr/ite-34). In 0.7, this will accidentally create a
directory datapoints-wr in the kafka log root dir. The problem has been
fixed in 0.8 by restricting the topic names to alpha-numeric plus "-" and
"_".

Thanks,

Jun
On Fri, Jul 19, 2013 at 7:45 AM, Blake Smith <[EMAIL PROTECTED]>wrote:

> Just to close the loop here for posterity:
>
> 1. For the directory topic name corruption, it looks like there's still an
> outstanding issue in JIRA: https://issues.apache.org/jira/browse/KAFKA-411
> 2. Ensuring log recovery is run seems to be fixed in
> commit 75fc5eab35aa33cffd9c09a2070dfe287db0ef4e (
> https://issues.apache.org/jira/browse/KAFKA-188)
>
> Thanks for your help Jay, it looks like we'll have to start the 0.7.1 ->
> 0.8 upgrade process sooner than I thought.
>
>
> On Wed, Jul 17, 2013 at 12:32 PM, Blake Smith <[EMAIL PROTECTED]
> >wrote:
>
> > 1. Cool. For now I'll write this off as a fluke side-effect of a unsafe
> > hardware shutdown. If I come up with a reproducible test case, I'll send
> it
> > (or a patch) your way.
> >
> > 2. After the process came up after the crash, recovery tried to run on
> > each partition, but because of the malformed directory issue it looks
> like
> > recovery never ran on partition 11 (the log with the corrupt message).
> >
> > 40472 [main] INFO  kafka.log.Log  - Loading the last segment
> > /var/kafka/datapoints-write-33/00000000093417473922.kafka in mutable
> mode,
> > recovery true
> > 44032 [main] INFO  kafka.message.FileMessageSet  - recover high water
> > mark:199897864
> > 44032 [main] INFO  kafka.message.FileMessageSet  - Recovery succeeded in
> 3
> > seconds. 0 bytes truncated.
> > 44033 [main] INFO  kafka.log.LogManager  - Loading log
> > 'datapoints-write-35'
> > 44044 [main] INFO  kafka.log.Log  - Loading the last segment
> > /var/kafka/datapoints-write-35/00000000092880891040.kafka in mutable
> mode,
> > recovery true
> > 48836 [main] INFO  kafka.message.FileMessageSet  - recover high water
> > mark:268123698
> > 48836 [main] INFO  kafka.message.FileMessageSet  - Recovery succeeded in
> 4
> > seconds. 0 bytes truncated.
> > 48859 [main] INFO  kafka.log.LogManager  - Loading log 'datapoints-wr'
> > 48911 [main] FATAL kafka.server.KafkaServerStartable  - Fatal error
> during
> > KafkaServerStable startup. Prepare to shutdown
> > java.lang.NumberFormatException: For input string: "wr"
> >         at
> >
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> >         at java.lang.Integer.parseInt(Integer.java:481)
> >         at java.lang.Integer.parseInt(Integer.java:514)
> >         at
> > scala.collection.immutable.StringLike$class.toInt(StringLike.scala:207)
> >         at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
> >         at kafka.utils.Utils$.getTopicPartition(Utils.scala:558)
> >         at kafka.log.LogManager$$anonfun$4.apply(LogManager.scala:71)
> >         at kafka.log.LogManager$$anonfun$4.apply(LogManager.scala:65)
> >         at
> >
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
> >         at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
> >         at kafka.log.LogManager.<init>(LogManager.scala:65)
> >         at kafka.server.KafkaServer.startup(KafkaServer.scala:58)
> >         at
> > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> >         at kafka.Kafka$.main(Kafka.scala:50)
> >         at kafka.Kafka.main(Kafka.scala)
> > 48912 [main] INFO  kafka.server.KafkaServer  - Shutting down Kafka server
> > 48913 [main] INFO  kafka.utils.KafkaScheduler  - shutdown scheduler
> > kafka-logcleaner-
> > 48913 [main] INFO  kafka.server.KafkaServer  - Kafka server shut down
> > completed
> > 0    [main] INFO  kafka.utils.Utils$  - The number of partitions for
> topic
> >  datapoints-write : 50
> > 2    [main] INFO  kafka.utils.Utils$  - The number of partitions for

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB