Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Hardware profile


+
Ian Friedman 2013-03-29, 15:33
+
Jun Rao 2013-03-29, 16:44
+
Ian Friedman 2013-03-29, 18:38
+
Neha Narkhede 2013-03-29, 18:46
+
David Arthur 2013-03-29, 18:48
+
Jun Rao 2013-03-29, 22:54
+
David Arthur 2013-03-29, 23:44
Copy link to this message
-
Re: Hardware profile
Neha Narkhede 2013-03-29, 23:54
gigabit ethernet

Thanks,
Neha

On Fri, Mar 29, 2013 at 4:44 PM, David Arthur <[EMAIL PROTECTED]> wrote:
> Especially in light of replication (broker-broker communication), I'm
> wondering if all the brokers are in the same rack and what kind of
> networking interfaces are used (Gigabit ethernet, Fibre Channel, etc).
>
>
> On 3/29/13 6:53 PM, Jun Rao wrote:
>>
>> We have multiple Kafka clusters, each has about 10 brokers right now. Not
>> sure about the network topology. What kind of info do you want to know?
>>
>> Thanks,
>>
>> Jun
>>
>> On Fri, Mar 29, 2013 at 11:47 AM, David Arthur <[EMAIL PROTECTED]> wrote:
>>
>>> How many brokers are you (LinkedIn) running? What kind of network
>>> topology?
>>>
>>>
>>> On 3/29/13 2:45 PM, Neha Narkhede wrote:
>>>
>>>> 1. We never share zookeeper and broker on the same hardware. Both need
>>>> significant memory to operate efficiently.
>>>> 2. 14 drive setup is just for Kafka. We have a separate disk for the OS,
>>>> AFAIK.
>>>>
>>>> Thanks,
>>>> Neha
>>>>
>>>> On Fri, Mar 29, 2013 at 11:37 AM, Ian Friedman <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Thanks Jun. Couple more questions:
>>>>> 1. Do you guys have dedicated hardware for Zookeeper or do you have a
>>>>> few machines run both a ZK and a broker? If so, do you keep the ZK and
>>>>> Kafka data on separate volumes?
>>>>> 2. You use the 14 drive raid setup is just for Kafka data and a
>>>>> separate
>>>>> drive for the OS?
>>>>>
>>>>> Thanks again,
>>>>> Ian
>>>>>
>>>>>
>>>>> On Friday, March 29, 2013 at 12:43 PM, Jun Rao wrote:
>>>>>
>>>>>   It's more or less the same. Our new server has 14 sata disks, each of
>>>>> 1
>>>>>>
>>>>>> TB.
>>>>>> The disk also has better write latency due to larger write cache.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Jun
>>>>>>
>>>>>> On Fri, Mar 29, 2013 at 8:32 AM, Ian Friedman <[EMAIL PROTECTED] (mailto:
>>>>>> [EMAIL PROTECTED])> wrote:
>>>>>>
>>>>>>   Hi all,
>>>>>>>
>>>>>>> I'm wondering how up to date the hardware specs listed on this page
>>>>>>> are:
>>>>>>>
>>>>>>> https://cwiki.apache.org/**confluence/display/KAFKA/**Operations<https://cwiki.apache.org/confluence/display/KAFKA/Operations>
>>>>>>>
>>>>>>> We're evaluating hardware for a Kafka broker/ZK quorum buildout and
>>>>>>> looking for some tips and/or sample configurations if anyone can help
>>>>>>> us
>>>>>>> out with some recommendations.
>>>>>>>
>>>>>>> Thanks in advance,
>>>>>>> Ian
>>>>>>>
>>>>>>>
>>>>>>
>