Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Copy kafka data between servers?


+
Jason Huang 2013-02-27, 10:11
+
Jun Rao 2013-02-27, 17:48
+
Neha Narkhede 2013-02-27, 17:58
+
Jason Huang 2013-02-27, 21:31
+
Neha Narkhede 2013-02-27, 23:59
Copy link to this message
-
Re: Copy kafka data between servers?
I've started by only coping $log.dir from server A to server B. Both
server A and server B ran same version of kafka 0.8 with same
configuration files.

However, after running kafka 0.8 on server B I get the following
exception when I tried to fetch the message:
2013-02-28 05:56:35,851] WARN [KafkaApi-1] Error while responding to
offset request (kafka.server.KafkaApis)
kafka.common.UnknownTopicOrPartitionException: Topic topic_general
partition 0 doesn't exist on 1
at kafka.server.ReplicaManager.getLeaderReplicaIfLocal(ReplicaManager.scala:163)........

However, the folder topic_general-0 exists and there are files
00000000000000000000.log and 00000000000000000000.index there . There
are also a replication-offset-checkpoint file in this $log.dir folder.
I then copied by $log.dir and also the zookeeper folder from server A
to server B and run it. In the zookeeper folder I have the following
files:
-rw-r--r--. 1 root root      296 Feb 28 06:12 snapshot.0
-rw-r--r--. 1 root root 67108880 Feb 28 06:12 log.1
-rw-r--r--. 1 root root 67108880 Feb 28 06:12 log.4b
-rw-r--r--. 1 root root     4817 Feb 28 06:12 snapshot.4a

With both log data and zookeeper data copied over to server B I am
getting start up errors in zookeeper log
INFO Got user-level KeeperException when processing
sessionid:0x13d20830d3e0000 type:create cxid:0x1
zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/brokers/ids/1 Error:KeeperErrorCode = NodeExists for
/brokers/ids/1 (org.apache.zookeeper.server.PrepRequestProcessor)

and start up errors in kafka server log:
[2013-02-28 06:15:00,228] ERROR [Partition state machine on Controller
1]: State change for partition
[topic_84784ecc-3803-42eb-bcdd-31dc42b697c6, 0] from OfflinePartition
to OnlinePartition failed (kafka.controller.PartitionStateMachine)
kafka.common.PartitionOfflineException: All replicas for partition
[topic_84784ecc-3803-42eb-bcdd-31dc42b697c6, 0] are dead. Marking this
partition offline

And I am getting the same error when trying to fetch messages:
2013-02-28 06:20:01,516] WARN [KafkaApi-1] Error while responding to
offset request (kafka.server.KafkaApis)
kafka.common.UnknownTopicOrPartitionException: Topic topic_general
partition 0 doesn't exist on 1
at kafka.server.ReplicaManager.getLeaderReplicaIfLocal(ReplicaManager.scala:163)

I am running both zookeeper and kafka on the same server.  I only have
one server so the replication factor is 1.

Looks like something went wrong for me. Any ideas?

thanks!

Jason

On Wed, Feb 27, 2013 at 6:59 PM, Neha Narkhede <[EMAIL PROTECTED]> wrote:
> As Jun described, the purpose of broker.id is to be able to move data from
> one broker to the other without changes. I believe this should work in 0.8
> as well. However, we've never tried it so not sure if there are bugs. Let
> us know how it goes.
>
> Thanks,
> Neha
>
>
> On Wed, Feb 27, 2013 at 1:31 PM, Jason Huang <[EMAIL PROTECTED]> wrote:
>
>> Thanks for the response.
>>
>> I apologize - I should have mentioned that we are trying out kafka 0.8
>> because of its new exciting features.
>>
>> So for 0.8, what do I need to do with the zookeeper data? copy
>> zookeeper file from $dataDir in zookeeper.properties from server A to
>> server B, and then run server B's kafka instance?
>>
>> Jason
>>
>> On Wed, Feb 27, 2013 at 12:57 PM, Neha Narkhede <[EMAIL PROTECTED]>
>> wrote:
>> > Jason,
>> >
>> > Assuming this question is for 0.7, you can just copy the data in the log
>> > directory and restart the new broker. You don't need to copy the
>> zookeeper
>> > data.
>> >
>> > Thanks,
>> > Neha
>> > On Feb 27, 2013 2:11 AM, "Jason Huang" <[EMAIL PROTECTED]> wrote:
>> >
>> >> Hello,
>> >>
>> >> Is there a way to migrate kafka data from server A to another server B?
>> >>
>> >> Can I just copy the data from $log.dir in server.properties and
>> >> zookeeper file from $dataDir in zookeeper.properties from server A to
>> >> server B, and then run server B's kafka instance?
>> >>
>> >> thanks,

 
+
Neha Narkhede 2013-02-28, 17:10
+
Jason Huang 2013-03-01, 01:11
+
Neha Narkhede 2013-03-01, 22:41
+
Jason Huang 2013-03-03, 11:44
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB