Kafka, mail # user - Re: Copy kafka data between servers? - 2013-02-28, 11:21
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Re: Copy kafka data between servers?
I've started by only coping $log.dir from server A to server B. Both
server A and server B ran same version of kafka 0.8 with same
configuration files.

However, after running kafka 0.8 on server B I get the following
exception when I tried to fetch the message:
2013-02-28 05:56:35,851] WARN [KafkaApi-1] Error while responding to
offset request (kafka.server.KafkaApis)
kafka.common.UnknownTopicOrPartitionException: Topic topic_general
partition 0 doesn't exist on 1
at kafka.server.ReplicaManager.getLeaderReplicaIfLocal(ReplicaManager.scala:163)........

However, the folder topic_general-0 exists and there are files
00000000000000000000.log and 00000000000000000000.index there . There
are also a replication-offset-checkpoint file in this $log.dir folder.
I then copied by $log.dir and also the zookeeper folder from server A
to server B and run it. In the zookeeper folder I have the following
files:
-rw-r--r--. 1 root root      296 Feb 28 06:12 snapshot.0
-rw-r--r--. 1 root root 67108880 Feb 28 06:12 log.1
-rw-r--r--. 1 root root 67108880 Feb 28 06:12 log.4b
-rw-r--r--. 1 root root     4817 Feb 28 06:12 snapshot.4a

With both log data and zookeeper data copied over to server B I am
getting start up errors in zookeeper log
INFO Got user-level KeeperException when processing
sessionid:0x13d20830d3e0000 type:create cxid:0x1
zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/brokers/ids/1 Error:KeeperErrorCode = NodeExists for
/brokers/ids/1 (org.apache.zookeeper.server.PrepRequestProcessor)

and start up errors in kafka server log:
[2013-02-28 06:15:00,228] ERROR [Partition state machine on Controller
1]: State change for partition
[topic_84784ecc-3803-42eb-bcdd-31dc42b697c6, 0] from OfflinePartition
to OnlinePartition failed (kafka.controller.PartitionStateMachine)
kafka.common.PartitionOfflineException: All replicas for partition
[topic_84784ecc-3803-42eb-bcdd-31dc42b697c6, 0] are dead. Marking this
partition offline

And I am getting the same error when trying to fetch messages:
2013-02-28 06:20:01,516] WARN [KafkaApi-1] Error while responding to
offset request (kafka.server.KafkaApis)
kafka.common.UnknownTopicOrPartitionException: Topic topic_general
partition 0 doesn't exist on 1
at kafka.server.ReplicaManager.getLeaderReplicaIfLocal(ReplicaManager.scala:163)

I am running both zookeeper and kafka on the same server.  I only have
one server so the replication factor is 1.

Looks like something went wrong for me. Any ideas?

thanks!

Jason

On Wed, Feb 27, 2013 at 6:59 PM, Neha Narkhede <[EMAIL PROTECTED]> wrote:

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB