howard chen
2012-10-30, 04:07
Jun Rao
2012-10-30, 04:27
howard chen
2012-10-30, 04:34
Jun Rao
2012-10-30, 13:21
Matthew Rathbone
2012-10-30, 15:07
Neha Narkhede
2012-10-30, 16:46
howard chen
2012-10-31, 14:35
Jay Kreps
2012-10-31, 15:40
howard chen
2012-11-01, 03:46
-
How large memory I needed to the zookeeper and kafka server?howard chen 2012-10-30, 04:07
I understand the main limitation of Kafka deployment is the disk space,
E.g.
If I generate 10GB message per day, and I have 2 nodes, and I need to
keep for 10 days, then I need
10GB * 10 / 2 = 50GB per node (of course there are overhead, but the
requirement is somehow proportional.)
So If I deploy machinesusing the following setup, do you think they
are reasonable?
2 x Kafka (100GB, 2xCPU x 2GB RAM)
3 x Zookeeper ( 10GB, 1x CPU, 512MB RAM)
do you think it is okay?
E.g.
If I generate 10GB message per day, and I have 2 nodes, and I need to
keep for 10 days, then I need
10GB * 10 / 2 = 50GB per node (of course there are overhead, but the
requirement is somehow proportional.)
So If I deploy machinesusing the following setup, do you think they
are reasonable?
2 x Kafka (100GB, 2xCPU x 2GB RAM)
3 x Zookeeper ( 10GB, 1x CPU, 512MB RAM)
do you think it is okay?
-
Re: How large memory I needed to the zookeeper and kafka server?Jun Rao 2012-10-30, 04:27
Kafka broker typically doesn't need a lot memory. So 2GB is fine. ZK memory
depends on # consumers. More consumers mean more offsets written to ZK.
Thanks,
Jun
On Mon, Oct 29, 2012 at 9:07 PM, howard chen <[EMAIL PROTECTED]> wrote:
> I understand the main limitation of Kafka deployment is the disk space,
>
> E.g.
>
> If I generate 10GB message per day, and I have 2 nodes, and I need to
> keep for 10 days, then I need
>
> 10GB * 10 / 2 = 50GB per node (of course there are overhead, but the
> requirement is somehow proportional.)
>
> So If I deploy machinesusing the following setup, do you think they
> are reasonable?
>
> 2 x Kafka (100GB, 2xCPU x 2GB RAM)
> 3 x Zookeeper ( 10GB, 1x CPU, 512MB RAM)
>
> do you think it is okay?
>
depends on # consumers. More consumers mean more offsets written to ZK.
Thanks,
Jun
On Mon, Oct 29, 2012 at 9:07 PM, howard chen <[EMAIL PROTECTED]> wrote:
> I understand the main limitation of Kafka deployment is the disk space,
>
> E.g.
>
> If I generate 10GB message per day, and I have 2 nodes, and I need to
> keep for 10 days, then I need
>
> 10GB * 10 / 2 = 50GB per node (of course there are overhead, but the
> requirement is somehow proportional.)
>
> So If I deploy machinesusing the following setup, do you think they
> are reasonable?
>
> 2 x Kafka (100GB, 2xCPU x 2GB RAM)
> 3 x Zookeeper ( 10GB, 1x CPU, 512MB RAM)
>
> do you think it is okay?
>
-
Re: How large memory I needed to the zookeeper and kafka server?howard chen 2012-10-30, 04:34
We only a few internal consumers, so assume they should be fine?
On Tue, Oct 30, 2012 at 12:27 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> Kafka broker typically doesn't need a lot memory. So 2GB is fine. ZK memory
> depends on # consumers. More consumers mean more offsets written to ZK.
>
> Thanks,
>
> Jun
>
> On Mon, Oct 29, 2012 at 9:07 PM, howard chen <[EMAIL PROTECTED]> wrote:
>
>> I understand the main limitation of Kafka deployment is the disk space,
>>
>> E.g.
>>
>> If I generate 10GB message per day, and I have 2 nodes, and I need to
>> keep for 10 days, then I need
>>
>> 10GB * 10 / 2 = 50GB per node (of course there are overhead, but the
>> requirement is somehow proportional.)
>>
>> So If I deploy machinesusing the following setup, do you think they
>> are reasonable?
>>
>> 2 x Kafka (100GB, 2xCPU x 2GB RAM)
>> 3 x Zookeeper ( 10GB, 1x CPU, 512MB RAM)
>>
>> do you think it is okay?
>>
On Tue, Oct 30, 2012 at 12:27 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> Kafka broker typically doesn't need a lot memory. So 2GB is fine. ZK memory
> depends on # consumers. More consumers mean more offsets written to ZK.
>
> Thanks,
>
> Jun
>
> On Mon, Oct 29, 2012 at 9:07 PM, howard chen <[EMAIL PROTECTED]> wrote:
>
>> I understand the main limitation of Kafka deployment is the disk space,
>>
>> E.g.
>>
>> If I generate 10GB message per day, and I have 2 nodes, and I need to
>> keep for 10 days, then I need
>>
>> 10GB * 10 / 2 = 50GB per node (of course there are overhead, but the
>> requirement is somehow proportional.)
>>
>> So If I deploy machinesusing the following setup, do you think they
>> are reasonable?
>>
>> 2 x Kafka (100GB, 2xCPU x 2GB RAM)
>> 3 x Zookeeper ( 10GB, 1x CPU, 512MB RAM)
>>
>> do you think it is okay?
>>
-
Re: How large memory I needed to the zookeeper and kafka server?Jun Rao 2012-10-30, 13:21
Yes.
Thanks,
Jun
On Mon, Oct 29, 2012 at 9:34 PM, howard chen <[EMAIL PROTECTED]> wrote:
> We only a few internal consumers, so assume they should be fine?
>
>
> On Tue, Oct 30, 2012 at 12:27 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > Kafka broker typically doesn't need a lot memory. So 2GB is fine. ZK
> memory
> > depends on # consumers. More consumers mean more offsets written to ZK.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Oct 29, 2012 at 9:07 PM, howard chen <[EMAIL PROTECTED]> wrote:
> >
> >> I understand the main limitation of Kafka deployment is the disk space,
> >>
> >> E.g.
> >>
> >> If I generate 10GB message per day, and I have 2 nodes, and I need to
> >> keep for 10 days, then I need
> >>
> >> 10GB * 10 / 2 = 50GB per node (of course there are overhead, but the
> >> requirement is somehow proportional.)
> >>
> >> So If I deploy machinesusing the following setup, do you think they
> >> are reasonable?
> >>
> >> 2 x Kafka (100GB, 2xCPU x 2GB RAM)
> >> 3 x Zookeeper ( 10GB, 1x CPU, 512MB RAM)
> >>
> >> do you think it is okay?
> >>
>
Thanks,
Jun
On Mon, Oct 29, 2012 at 9:34 PM, howard chen <[EMAIL PROTECTED]> wrote:
> We only a few internal consumers, so assume they should be fine?
>
>
> On Tue, Oct 30, 2012 at 12:27 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > Kafka broker typically doesn't need a lot memory. So 2GB is fine. ZK
> memory
> > depends on # consumers. More consumers mean more offsets written to ZK.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Oct 29, 2012 at 9:07 PM, howard chen <[EMAIL PROTECTED]> wrote:
> >
> >> I understand the main limitation of Kafka deployment is the disk space,
> >>
> >> E.g.
> >>
> >> If I generate 10GB message per day, and I have 2 nodes, and I need to
> >> keep for 10 days, then I need
> >>
> >> 10GB * 10 / 2 = 50GB per node (of course there are overhead, but the
> >> requirement is somehow proportional.)
> >>
> >> So If I deploy machinesusing the following setup, do you think they
> >> are reasonable?
> >>
> >> 2 x Kafka (100GB, 2xCPU x 2GB RAM)
> >> 3 x Zookeeper ( 10GB, 1x CPU, 512MB RAM)
> >>
> >> do you think it is okay?
> >>
>
-
Re: How large memory I needed to the zookeeper and kafka server?Matthew Rathbone 2012-10-30, 15:07
When thinking about memory for the broker, the only thing you should
consider is the filesystem cache. The further behind production you're
consuming, the more memory matters (eg keeping your cache window larger
than the gap between production and consumption).
As a caveat, this is only important if you're really thrashing the brokers
hard, we don't even see a blip if we consume from disk and we're pushing
them as hard as we can :-).
On Tue, Oct 30, 2012 at 8:21 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> Yes.
>
> Thanks,
>
> Jun
>
> On Mon, Oct 29, 2012 at 9:34 PM, howard chen <[EMAIL PROTECTED]> wrote:
>
> > We only a few internal consumers, so assume they should be fine?
> >
> >
> > On Tue, Oct 30, 2012 at 12:27 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > Kafka broker typically doesn't need a lot memory. So 2GB is fine. ZK
> > memory
> > > depends on # consumers. More consumers mean more offsets written to ZK.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Oct 29, 2012 at 9:07 PM, howard chen <[EMAIL PROTECTED]>
> wrote:
> > >
> > >> I understand the main limitation of Kafka deployment is the disk
> space,
> > >>
> > >> E.g.
> > >>
> > >> If I generate 10GB message per day, and I have 2 nodes, and I need to
> > >> keep for 10 days, then I need
> > >>
> > >> 10GB * 10 / 2 = 50GB per node (of course there are overhead, but the
> > >> requirement is somehow proportional.)
> > >>
> > >> So If I deploy machinesusing the following setup, do you think they
> > >> are reasonable?
> > >>
> > >> 2 x Kafka (100GB, 2xCPU x 2GB RAM)
> > >> 3 x Zookeeper ( 10GB, 1x CPU, 512MB RAM)
> > >>
> > >> do you think it is okay?
> > >>
> >
>
--
Matthew Rathbone
Foursquare | Software Engineer | Server Engineering Team
[EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
4sq<http://foursquare.com/rathboma>
consider is the filesystem cache. The further behind production you're
consuming, the more memory matters (eg keeping your cache window larger
than the gap between production and consumption).
As a caveat, this is only important if you're really thrashing the brokers
hard, we don't even see a blip if we consume from disk and we're pushing
them as hard as we can :-).
On Tue, Oct 30, 2012 at 8:21 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> Yes.
>
> Thanks,
>
> Jun
>
> On Mon, Oct 29, 2012 at 9:34 PM, howard chen <[EMAIL PROTECTED]> wrote:
>
> > We only a few internal consumers, so assume they should be fine?
> >
> >
> > On Tue, Oct 30, 2012 at 12:27 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > Kafka broker typically doesn't need a lot memory. So 2GB is fine. ZK
> > memory
> > > depends on # consumers. More consumers mean more offsets written to ZK.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Oct 29, 2012 at 9:07 PM, howard chen <[EMAIL PROTECTED]>
> wrote:
> > >
> > >> I understand the main limitation of Kafka deployment is the disk
> space,
> > >>
> > >> E.g.
> > >>
> > >> If I generate 10GB message per day, and I have 2 nodes, and I need to
> > >> keep for 10 days, then I need
> > >>
> > >> 10GB * 10 / 2 = 50GB per node (of course there are overhead, but the
> > >> requirement is somehow proportional.)
> > >>
> > >> So If I deploy machinesusing the following setup, do you think they
> > >> are reasonable?
> > >>
> > >> 2 x Kafka (100GB, 2xCPU x 2GB RAM)
> > >> 3 x Zookeeper ( 10GB, 1x CPU, 512MB RAM)
> > >>
> > >> do you think it is okay?
> > >>
> >
>
--
Matthew Rathbone
Foursquare | Software Engineer | Server Engineering Team
[EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
4sq<http://foursquare.com/rathboma>
-
Re: How large memory I needed to the zookeeper and kafka server?Neha Narkhede 2012-10-30, 16:46
We've added our production deployment experience for Kafka and
Zookeeper here -
https://cwiki.apache.org/confluence/display/KAFKA/Operations
Thanks,
Neha
On Tue, Oct 30, 2012 at 8:07 AM, Matthew Rathbone
<[EMAIL PROTECTED]> wrote:
> When thinking about memory for the broker, the only thing you should
> consider is the filesystem cache. The further behind production you're
> consuming, the more memory matters (eg keeping your cache window larger
> than the gap between production and consumption).
>
> As a caveat, this is only important if you're really thrashing the brokers
> hard, we don't even see a blip if we consume from disk and we're pushing
> them as hard as we can :-).
>
>
>
> On Tue, Oct 30, 2012 at 8:21 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
>> Yes.
>>
>> Thanks,
>>
>> Jun
>>
>> On Mon, Oct 29, 2012 at 9:34 PM, howard chen <[EMAIL PROTECTED]> wrote:
>>
>> > We only a few internal consumers, so assume they should be fine?
>> >
>> >
>> > On Tue, Oct 30, 2012 at 12:27 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> > > Kafka broker typically doesn't need a lot memory. So 2GB is fine. ZK
>> > memory
>> > > depends on # consumers. More consumers mean more offsets written to ZK.
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > > On Mon, Oct 29, 2012 at 9:07 PM, howard chen <[EMAIL PROTECTED]>
>> wrote:
>> > >
>> > >> I understand the main limitation of Kafka deployment is the disk
>> space,
>> > >>
>> > >> E.g.
>> > >>
>> > >> If I generate 10GB message per day, and I have 2 nodes, and I need to
>> > >> keep for 10 days, then I need
>> > >>
>> > >> 10GB * 10 / 2 = 50GB per node (of course there are overhead, but the
>> > >> requirement is somehow proportional.)
>> > >>
>> > >> So If I deploy machinesusing the following setup, do you think they
>> > >> are reasonable?
>> > >>
>> > >> 2 x Kafka (100GB, 2xCPU x 2GB RAM)
>> > >> 3 x Zookeeper ( 10GB, 1x CPU, 512MB RAM)
>> > >>
>> > >> do you think it is okay?
>> > >>
>> >
>>
>
>
>
> --
> Matthew Rathbone
> Foursquare | Software Engineer | Server Engineering Team
> [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
> 4sq<http://foursquare.com/rathboma>
Zookeeper here -
https://cwiki.apache.org/confluence/display/KAFKA/Operations
Thanks,
Neha
On Tue, Oct 30, 2012 at 8:07 AM, Matthew Rathbone
<[EMAIL PROTECTED]> wrote:
> When thinking about memory for the broker, the only thing you should
> consider is the filesystem cache. The further behind production you're
> consuming, the more memory matters (eg keeping your cache window larger
> than the gap between production and consumption).
>
> As a caveat, this is only important if you're really thrashing the brokers
> hard, we don't even see a blip if we consume from disk and we're pushing
> them as hard as we can :-).
>
>
>
> On Tue, Oct 30, 2012 at 8:21 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
>> Yes.
>>
>> Thanks,
>>
>> Jun
>>
>> On Mon, Oct 29, 2012 at 9:34 PM, howard chen <[EMAIL PROTECTED]> wrote:
>>
>> > We only a few internal consumers, so assume they should be fine?
>> >
>> >
>> > On Tue, Oct 30, 2012 at 12:27 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> > > Kafka broker typically doesn't need a lot memory. So 2GB is fine. ZK
>> > memory
>> > > depends on # consumers. More consumers mean more offsets written to ZK.
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > > On Mon, Oct 29, 2012 at 9:07 PM, howard chen <[EMAIL PROTECTED]>
>> wrote:
>> > >
>> > >> I understand the main limitation of Kafka deployment is the disk
>> space,
>> > >>
>> > >> E.g.
>> > >>
>> > >> If I generate 10GB message per day, and I have 2 nodes, and I need to
>> > >> keep for 10 days, then I need
>> > >>
>> > >> 10GB * 10 / 2 = 50GB per node (of course there are overhead, but the
>> > >> requirement is somehow proportional.)
>> > >>
>> > >> So If I deploy machinesusing the following setup, do you think they
>> > >> are reasonable?
>> > >>
>> > >> 2 x Kafka (100GB, 2xCPU x 2GB RAM)
>> > >> 3 x Zookeeper ( 10GB, 1x CPU, 512MB RAM)
>> > >>
>> > >> do you think it is okay?
>> > >>
>> >
>>
>
>
>
> --
> Matthew Rathbone
> Foursquare | Software Engineer | Server Engineering Team
> [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
> 4sq<http://foursquare.com/rathboma>
-
Re: How large memory I needed to the zookeeper and kafka server?howard chen 2012-10-31, 14:35
Hi,
On Tue, Oct 30, 2012 at 11:07 PM, Matthew Rathbone
<[EMAIL PROTECTED]> wrote:
> When thinking about memory for the broker, the only thing you should
> consider is the filesystem cache.
So you mean if I have 4GB ram, I should leave some memory for the OS
(i.e. file system cache), instead of allocating all of them to the
JVM?
On Tue, Oct 30, 2012 at 11:07 PM, Matthew Rathbone
<[EMAIL PROTECTED]> wrote:
> When thinking about memory for the broker, the only thing you should
> consider is the filesystem cache.
So you mean if I have 4GB ram, I should leave some memory for the OS
(i.e. file system cache), instead of allocating all of them to the
JVM?
-
Re: How large memory I needed to the zookeeper and kafka server?Jay Kreps 2012-10-31, 15:40
Yes. Kafka doesn't use much in process memory, our 4GB is probably just
because no one bothered to tune it. But memory given to the JVM takes away
from the OS cache and will result in more disk I/O.
-Jay
On Wed, Oct 31, 2012 at 7:35 AM, howard chen <[EMAIL PROTECTED]> wrote:
> Hi,
>
>
> On Tue, Oct 30, 2012 at 11:07 PM, Matthew Rathbone
> <[EMAIL PROTECTED]> wrote:
> > When thinking about memory for the broker, the only thing you should
> > consider is the filesystem cache.
>
> So you mean if I have 4GB ram, I should leave some memory for the OS
> (i.e. file system cache), instead of allocating all of them to the
> JVM?
>
because no one bothered to tune it. But memory given to the JVM takes away
from the OS cache and will result in more disk I/O.
-Jay
On Wed, Oct 31, 2012 at 7:35 AM, howard chen <[EMAIL PROTECTED]> wrote:
> Hi,
>
>
> On Tue, Oct 30, 2012 at 11:07 PM, Matthew Rathbone
> <[EMAIL PROTECTED]> wrote:
> > When thinking about memory for the broker, the only thing you should
> > consider is the filesystem cache.
>
> So you mean if I have 4GB ram, I should leave some memory for the OS
> (i.e. file system cache), instead of allocating all of them to the
> JVM?
>
-
Re: How large memory I needed to the zookeeper and kafka server?howard chen 2012-11-01, 03:46
Hi,
On Wed, Oct 31, 2012 at 11:40 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> Yes. Kafka doesn't use much in process memory, our 4GB is probably just
> because no one bothered to tune it. But memory given to the JVM takes away
> from the OS cache and will result in more disk I/O.
>
So if we allocate 50% of memory to Kafka (e.g. 1GB out of a 2GB box),
sound reasonable?
As it is JVM, OOM is painful to deal with..
thanks.
On Wed, Oct 31, 2012 at 11:40 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> Yes. Kafka doesn't use much in process memory, our 4GB is probably just
> because no one bothered to tune it. But memory given to the JVM takes away
> from the OS cache and will result in more disk I/O.
>
So if we allocate 50% of memory to Kafka (e.g. 1GB out of a 2GB box),
sound reasonable?
As it is JVM, OOM is painful to deal with..
thanks.
