Hi all,

I am running kafka-0.7.0 and found that even though expired segments are
deleted by broker, the disk space is not released. So, du shows that a
partition is 450 GB, but the total size of the files in the partition is
only about 250GB. Is this a bug? We are using raid10.

Thanks,

-Xiaoyu

Xiaoyu,

Please could you paste the log statements that specify which files are
deleted by the broker and also the detailed output by du on the Kafka
log directory ?
If the broker says it has garbage collected some log segments, space
should free up.

Thanks,
Neha

On Wed, Aug 8, 2012 at 12:22 PM, xiaoyu wang <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> I am running kafka-0.7.0 and found that even though expired segments are
> deleted by broker, the disk space is not released. So, du shows that a
> partition is 450 GB, but the total size of the files in the partition is
> only about 250GB. Is this a bug? We are using raid10.
>
> Thanks,
>
> -Xiaoyu

Neha,

I have attached the du and ls output for one of the partitions. Looks like
for some segments, du is showing 1GB while ls shows 513MB. Let me know if
you need additional info.

Thanks,

-Xiaoyu
On Wed, Aug 8, 2012 at 12:29 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Xiaoyu,
>
> Please could you paste the log statements that specify which files are
> deleted by the broker and also the detailed output by du on the Kafka
> log directory ?
> If the broker says it has garbage collected some log segments, space
> should free up.
>
> Thanks,
> Neha
>
> On Wed, Aug 8, 2012 at 12:22 PM, xiaoyu wang <[EMAIL PROTECTED]>
> wrote:
> > Hi all,
> >
> > I am running kafka-0.7.0 and found that even though expired segments are
> > deleted by broker, the disk space is not released. So, du shows that a
> > partition is 450 GB, but the total size of the files in the partition is
> > only about 250GB. Is this a bug? We are using raid10.
> >
> > Thanks,
> >
> > -Xiaoyu
>

The mailing list does not accept attachments - you will need to either
paste here or externally, say on gist.

Thanks,

Joel

On Wed, Aug 8, 2012 at 2:52 PM, xiaoyu wang <[EMAIL PROTECTED]> wrote:

> Neha,
>
> I have attached the du and ls output for one of the partitions. Looks like
> for some segments, du is showing 1GB while ls shows 513MB. Let me know if
> you need additional info.
>
> Thanks,
>
> -Xiaoyu
>
>
>
> On Wed, Aug 8, 2012 at 12:29 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
>
>> Xiaoyu,
>>
>> Please could you paste the log statements that specify which files are
>> deleted by the broker and also the detailed output by du on the Kafka
>> log directory ?
>> If the broker says it has garbage collected some log segments, space
>> should free up.
>>
>> Thanks,
>> Neha
>>
>> On Wed, Aug 8, 2012 at 12:22 PM, xiaoyu wang <[EMAIL PROTECTED]>
>> wrote:
>> > Hi all,
>> >
>> > I am running kafka-0.7.0 and found that even though expired segments are
>> > deleted by broker, the disk space is not released. So, du shows that a
>> > partition is 450 GB, but the total size of the files in the partition is
>> > only about 250GB. Is this a bug? We are using raid10.
>> >
>> > Thanks,
>> >
>> > -Xiaoyu
>>
>
>

Here is a sample. The highlighted lines are those showing different sizes
in du and ls

*du output
*
513M    ./00000001197773741184.kafka
1.0G    ./00000001200458096891.kafka
1.0G    ./00000001200994968784.kafka
1.0G    ./00000001201531839884.kafka
1.0G    ./00000001202068711094.kafka
513M    ./00000001011479389046.kafka

*ls output
*
-rw-r--r--  1 kafka kafka 536871053 Aug  8 13:01 00000001197773741184.kafka
-rw-r--r--  1 kafka kafka 536871893 Aug  8 13:36 00000001200458096891.kafka
-rw-r--r--  1 kafka kafka 536871100 Aug  8 13:45 00000001200994968784.kafka
-rw-r--r--  1 kafka kafka 536871210 Aug  8 13:49 00000001201531839884.kafka
-rw-r--r--  1 kafka kafka 536871063 Aug  8 14:00 00000001202068711094.kafka
-rw-r--r--  1 kafka kafka 536871139 Aug  6 07:02 00000001011479389046.kafka

On Wed, Aug 8, 2012 at 2:58 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:

> The mailing list does not accept attachments - you will need to either
> paste here or externally, say on gist.
>
> Thanks,
>
> Joel
>
> On Wed, Aug 8, 2012 at 2:52 PM, xiaoyu wang <[EMAIL PROTECTED]> wrote:
>
> > Neha,
> >
> > I have attached the du and ls output for one of the partitions. Looks
> like
> > for some segments, du is showing 1GB while ls shows 513MB. Let me know if
> > you need additional info.
> >
> > Thanks,
> >
> > -Xiaoyu
> >
> >
> >
> > On Wed, Aug 8, 2012 at 12:29 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
> >
> >> Xiaoyu,
> >>
> >> Please could you paste the log statements that specify which files are
> >> deleted by the broker and also the detailed output by du on the Kafka
> >> log directory ?
> >> If the broker says it has garbage collected some log segments, space
> >> should free up.
> >>
> >> Thanks,
> >> Neha
> >>
> >> On Wed, Aug 8, 2012 at 12:22 PM, xiaoyu wang <[EMAIL PROTECTED]>
> >> wrote:
> >> > Hi all,
> >> >
> >> > I am running kafka-0.7.0 and found that even though expired segments
> are
> >> > deleted by broker, the disk space is not released. So, du shows that a
> >> > partition is 450 GB, but the total size of the files in the partition
> is
> >> > only about 250GB. Is this a bug? We are using raid10.
> >> >
> >> > Thanks,
> >> >
> >> > -Xiaoyu
> >>
> >
> >
>

Xiaoyu,

Thanks for sending these! However, could you also send the Kafka log
that says these segments are deleted ?

Thanks,
Neha

On Wed, Aug 8, 2012 at 3:17 PM, xiaoyu wang <[EMAIL PROTECTED]> wrote:
> Here is a sample. The highlighted lines are those showing different sizes
> in du and ls
>
> *du output
> *
> 513M    ./00000001197773741184.kafka
> 1.0G    ./00000001200458096891.kafka
> 1.0G    ./00000001200994968784.kafka
> 1.0G    ./00000001201531839884.kafka
> 1.0G    ./00000001202068711094.kafka
> 513M    ./00000001011479389046.kafka
>
> *ls output
> *
> -rw-r--r--  1 kafka kafka 536871053 Aug  8 13:01 00000001197773741184.kafka
> -rw-r--r--  1 kafka kafka 536871893 Aug  8 13:36 00000001200458096891.kafka
> -rw-r--r--  1 kafka kafka 536871100 Aug  8 13:45 00000001200994968784.kafka
> -rw-r--r--  1 kafka kafka 536871210 Aug  8 13:49 00000001201531839884.kafka
> -rw-r--r--  1 kafka kafka 536871063 Aug  8 14:00 00000001202068711094.kafka
> -rw-r--r--  1 kafka kafka 536871139 Aug  6 07:02 00000001011479389046.kafka
>
> On Wed, Aug 8, 2012 at 2:58 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
>
>> The mailing list does not accept attachments - you will need to either
>> paste here or externally, say on gist.
>>
>> Thanks,
>>
>> Joel
>>
>> On Wed, Aug 8, 2012 at 2:52 PM, xiaoyu wang <[EMAIL PROTECTED]> wrote:
>>
>> > Neha,
>> >
>> > I have attached the du and ls output for one of the partitions. Looks
>> like
>> > for some segments, du is showing 1GB while ls shows 513MB. Let me know if
>> > you need additional info.
>> >
>> > Thanks,
>> >
>> > -Xiaoyu
>> >
>> >
>> >
>> > On Wed, Aug 8, 2012 at 12:29 PM, Neha Narkhede <[EMAIL PROTECTED]
>> >wrote:
>> >
>> >> Xiaoyu,
>> >>
>> >> Please could you paste the log statements that specify which files are
>> >> deleted by the broker and also the detailed output by du on the Kafka
>> >> log directory ?
>> >> If the broker says it has garbage collected some log segments, space
>> >> should free up.
>> >>
>> >> Thanks,
>> >> Neha
>> >>
>> >> On Wed, Aug 8, 2012 at 12:22 PM, xiaoyu wang <[EMAIL PROTECTED]>
>> >> wrote:
>> >> > Hi all,
>> >> >
>> >> > I am running kafka-0.7.0 and found that even though expired segments
>> are
>> >> > deleted by broker, the disk space is not released. So, du shows that a
>> >> > partition is 450 GB, but the total size of the files in the partition
>> is
>> >> > only about 250GB. Is this a bug? We are using raid10.
>> >> >
>> >> > Thanks,
>> >> >
>> >> > -Xiaoyu
>> >>
>> >
>> >
>>

Hmm, this seems like an issue with du.

Thanks

Jun

On Wed, Aug 8, 2012 at 3:17 PM, xiaoyu wang <[EMAIL PROTECTED]> wrote:

> Here is a sample. The highlighted lines are those showing different sizes
> in du and ls
>
> *du output
> *
> 513M    ./00000001197773741184.kafka
> 1.0G    ./00000001200458096891.kafka
> 1.0G    ./00000001200994968784.kafka
> 1.0G    ./00000001201531839884.kafka
> 1.0G    ./00000001202068711094.kafka
> 513M    ./00000001011479389046.kafka
>
> *ls output
> *
> -rw-r--r--  1 kafka kafka 536871053 Aug  8 13:01 00000001197773741184.kafka
> -rw-r--r--  1 kafka kafka 536871893 Aug  8 13:36 00000001200458096891.kafka
> -rw-r--r--  1 kafka kafka 536871100 Aug  8 13:45 00000001200994968784.kafka
> -rw-r--r--  1 kafka kafka 536871210 Aug  8 13:49 00000001201531839884.kafka
> -rw-r--r--  1 kafka kafka 536871063 Aug  8 14:00 00000001202068711094.kafka
> -rw-r--r--  1 kafka kafka 536871139 Aug  6 07:02 00000001011479389046.kafka
>
> On Wed, Aug 8, 2012 at 2:58 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
>
> > The mailing list does not accept attachments - you will need to either
> > paste here or externally, say on gist.
> >
> > Thanks,
> >
> > Joel
> >
> > On Wed, Aug 8, 2012 at 2:52 PM, xiaoyu wang <[EMAIL PROTECTED]>
> wrote:
> >
> > > Neha,
> > >
> > > I have attached the du and ls output for one of the partitions. Looks
> > like
> > > for some segments, du is showing 1GB while ls shows 513MB. Let me know
> if
> > > you need additional info.
> > >
> > > Thanks,
> > >
> > > -Xiaoyu
> > >
> > >
> > >
> > > On Wed, Aug 8, 2012 at 12:29 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > >wrote:
> > >
> > >> Xiaoyu,
> > >>
> > >> Please could you paste the log statements that specify which files are
> > >> deleted by the broker and also the detailed output by du on the Kafka
> > >> log directory ?
> > >> If the broker says it has garbage collected some log segments, space
> > >> should free up.
> > >>
> > >> Thanks,
> > >> Neha
> > >>
> > >> On Wed, Aug 8, 2012 at 12:22 PM, xiaoyu wang <[EMAIL PROTECTED]>
> > >> wrote:
> > >> > Hi all,
> > >> >
> > >> > I am running kafka-0.7.0 and found that even though expired segments
> > are
> > >> > deleted by broker, the disk space is not released. So, du shows
> that a
> > >> > partition is 450 GB, but the total size of the files in the
> partition
> > is
> > >> > only about 250GB. Is this a bug? We are using raid10.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > -Xiaoyu
> > >>
> > >
> > >
> >
>

Neha,

These are not deleted segments. Actually, most of them are new segments.
Because the total disk usage shown with "du" is smaller than "ls", I
thought it was because space not released for those segments that have
expired. But the detailed du output indicates that it is those active
segments that uses more disk spaces (shown 1GB in du but .5GB with ls).
 -Xiaoyu

On Wed, Aug 8, 2012 at 3:31 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Xiaoyu,
>
> Thanks for sending these! However, could you also send the Kafka log
> that says these segments are deleted ?
>
> Thanks,
> Neha
>
> On Wed, Aug 8, 2012 at 3:17 PM, xiaoyu wang <[EMAIL PROTECTED]> wrote:
> > Here is a sample. The highlighted lines are those showing different sizes
> > in du and ls
> >
> > *du output
> > *
> > 513M    ./00000001197773741184.kafka
> > 1.0G    ./00000001200458096891.kafka
> > 1.0G    ./00000001200994968784.kafka
> > 1.0G    ./00000001201531839884.kafka
> > 1.0G    ./00000001202068711094.kafka
> > 513M    ./00000001011479389046.kafka
> >
> > *ls output
> > *
> > -rw-r--r--  1 kafka kafka 536871053 Aug  8 13:01
> 00000001197773741184.kafka
> > -rw-r--r--  1 kafka kafka 536871893 Aug  8 13:36
> 00000001200458096891.kafka
> > -rw-r--r--  1 kafka kafka 536871100 Aug  8 13:45
> 00000001200994968784.kafka
> > -rw-r--r--  1 kafka kafka 536871210 Aug  8 13:49
> 00000001201531839884.kafka
> > -rw-r--r--  1 kafka kafka 536871063 Aug  8 14:00
> 00000001202068711094.kafka
> > -rw-r--r--  1 kafka kafka 536871139 Aug  6 07:02
> 00000001011479389046.kafka
> >
> > On Wed, Aug 8, 2012 at 2:58 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
> >
> >> The mailing list does not accept attachments - you will need to either
> >> paste here or externally, say on gist.
> >>
> >> Thanks,
> >>
> >> Joel
> >>
> >> On Wed, Aug 8, 2012 at 2:52 PM, xiaoyu wang <[EMAIL PROTECTED]>
> wrote:
> >>
> >> > Neha,
> >> >
> >> > I have attached the du and ls output for one of the partitions. Looks
> >> like
> >> > for some segments, du is showing 1GB while ls shows 513MB. Let me
> know if
> >> > you need additional info.
> >> >
> >> > Thanks,
> >> >
> >> > -Xiaoyu
> >> >
> >> >
> >> >
> >> > On Wed, Aug 8, 2012 at 12:29 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> >> >wrote:
> >> >
> >> >> Xiaoyu,
> >> >>
> >> >> Please could you paste the log statements that specify which files
> are
> >> >> deleted by the broker and also the detailed output by du on the Kafka
> >> >> log directory ?
> >> >> If the broker says it has garbage collected some log segments, space
> >> >> should free up.
> >> >>
> >> >> Thanks,
> >> >> Neha
> >> >>
> >> >> On Wed, Aug 8, 2012 at 12:22 PM, xiaoyu wang <[EMAIL PROTECTED]>
> >> >> wrote:
> >> >> > Hi all,
> >> >> >
> >> >> > I am running kafka-0.7.0 and found that even though expired
> segments
> >> are
> >> >> > deleted by broker, the disk space is not released. So, du shows
> that a
> >> >> > partition is 450 GB, but the total size of the files in the
> partition
> >> is
> >> >> > only about 250GB. Is this a bug? We are using raid10.
> >> >> >
> >> >> > Thanks,
> >> >> >
> >> >> > -Xiaoyu
> >> >>
> >> >
> >> >
> >>
>

Jun,

this does not seem to be a du problem because after restarting brokers, du
and ls show the same file size.

On Wed, Aug 8, 2012 at 3:55 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Hmm, this seems like an issue with du.
>
> Thanks
>
> Jun
>
> On Wed, Aug 8, 2012 at 3:17 PM, xiaoyu wang <[EMAIL PROTECTED]> wrote:
>
> > Here is a sample. The highlighted lines are those showing different sizes
> > in du and ls
> >
> > *du output
> > *
> > 513M    ./00000001197773741184.kafka
> > 1.0G    ./00000001200458096891.kafka
> > 1.0G    ./00000001200994968784.kafka
> > 1.0G    ./00000001201531839884.kafka
> > 1.0G    ./00000001202068711094.kafka
> > 513M    ./00000001011479389046.kafka
> >
> > *ls output
> > *
> > -rw-r--r--  1 kafka kafka 536871053 Aug  8 13:01
> 00000001197773741184.kafka
> > -rw-r--r--  1 kafka kafka 536871893 Aug  8 13:36
> 00000001200458096891.kafka
> > -rw-r--r--  1 kafka kafka 536871100 Aug  8 13:45
> 00000001200994968784.kafka
> > -rw-r--r--  1 kafka kafka 536871210 Aug  8 13:49
> 00000001201531839884.kafka
> > -rw-r--r--  1 kafka kafka 536871063 Aug  8 14:00
> 00000001202068711094.kafka
> > -rw-r--r--  1 kafka kafka 536871139 Aug  6 07:02
> 00000001011479389046.kafka
> >
> > On Wed, Aug 8, 2012 at 2:58 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
> >
> > > The mailing list does not accept attachments - you will need to either
> > > paste here or externally, say on gist.
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> > > On Wed, Aug 8, 2012 at 2:52 PM, xiaoyu wang <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Neha,
> > > >
> > > > I have attached the du and ls output for one of the partitions. Looks
> > > like
> > > > for some segments, du is showing 1GB while ls shows 513MB. Let me
> know
> > if
> > > > you need additional info.
> > > >
> > > > Thanks,
> > > >
> > > > -Xiaoyu
> > > >
> > > >
> > > >
> > > > On Wed, Aug 8, 2012 at 12:29 PM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > > >wrote:
> > > >
> > > >> Xiaoyu,
> > > >>
> > > >> Please could you paste the log statements that specify which files
> are
> > > >> deleted by the broker and also the detailed output by du on the
> Kafka
> > > >> log directory ?
> > > >> If the broker says it has garbage collected some log segments, space
> > > >> should free up.
> > > >>
> > > >> Thanks,
> > > >> Neha
> > > >>
> > > >> On Wed, Aug 8, 2012 at 12:22 PM, xiaoyu wang <[EMAIL PROTECTED]
> >
> > > >> wrote:
> > > >> > Hi all,
> > > >> >
> > > >> > I am running kafka-0.7.0 and found that even though expired
> segments
> > > are
> > > >> > deleted by broker, the disk space is not released. So, du shows
> > that a
> > > >> > partition is 450 GB, but the total size of the files in the
> > partition
> > > is
> > > >> > only about 250GB. Is this a bug? We are using raid10.
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > -Xiaoyu
> > > >>
> > > >
> > > >
> > >
> >
>

Does kafka pre-allocate 1GB for each segment? It shows that these segment
files are all open and used by kafka broker.

On Wed, Aug 8, 2012 at 4:14 PM, xiaoyu wang <[EMAIL PROTECTED]> wrote:

> Jun,
>
> this does not seem to be a du problem because after restarting brokers, du
> and ls show the same file size.
>
>
> On Wed, Aug 8, 2012 at 3:55 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
>> Hmm, this seems like an issue with du.
>>
>> Thanks
>>
>> Jun
>>
>> On Wed, Aug 8, 2012 at 3:17 PM, xiaoyu wang <[EMAIL PROTECTED]>
>> wrote:
>>
>> > Here is a sample. The highlighted lines are those showing different
>> sizes
>> > in du and ls
>> >
>> > *du output
>> > *
>> > 513M    ./00000001197773741184.kafka
>> > 1.0G    ./00000001200458096891.kafka
>> > 1.0G    ./00000001200994968784.kafka
>> > 1.0G    ./00000001201531839884.kafka
>> > 1.0G    ./00000001202068711094.kafka
>> > 513M    ./00000001011479389046.kafka
>> >
>> > *ls output
>> > *
>> > -rw-r--r--  1 kafka kafka 536871053 Aug  8 13:01
>> 00000001197773741184.kafka
>> > -rw-r--r--  1 kafka kafka 536871893 Aug  8 13:36
>> 00000001200458096891.kafka
>> > -rw-r--r--  1 kafka kafka 536871100 Aug  8 13:45
>> 00000001200994968784.kafka
>> > -rw-r--r--  1 kafka kafka 536871210 Aug  8 13:49
>> 00000001201531839884.kafka
>> > -rw-r--r--  1 kafka kafka 536871063 Aug  8 14:00
>> 00000001202068711094.kafka
>> > -rw-r--r--  1 kafka kafka 536871139 Aug  6 07:02
>> 00000001011479389046.kafka
>> >
>> > On Wed, Aug 8, 2012 at 2:58 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
>> >
>> > > The mailing list does not accept attachments - you will need to either
>> > > paste here or externally, say on gist.
>> > >
>> > > Thanks,
>> > >
>> > > Joel
>> > >
>> > > On Wed, Aug 8, 2012 at 2:52 PM, xiaoyu wang <[EMAIL PROTECTED]>
>> > wrote:
>> > >
>> > > > Neha,
>> > > >
>> > > > I have attached the du and ls output for one of the partitions.
>> Looks
>> > > like
>> > > > for some segments, du is showing 1GB while ls shows 513MB. Let me
>> know
>> > if
>> > > > you need additional info.
>> > > >
>> > > > Thanks,
>> > > >
>> > > > -Xiaoyu
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Aug 8, 2012 at 12:29 PM, Neha Narkhede <
>> > [EMAIL PROTECTED]
>> > > >wrote:
>> > > >
>> > > >> Xiaoyu,
>> > > >>
>> > > >> Please could you paste the log statements that specify which files
>> are
>> > > >> deleted by the broker and also the detailed output by du on the
>> Kafka
>> > > >> log directory ?
>> > > >> If the broker says it has garbage collected some log segments,
>> space
>> > > >> should free up.
>> > > >>
>> > > >> Thanks,
>> > > >> Neha
>> > > >>
>> > > >> On Wed, Aug 8, 2012 at 12:22 PM, xiaoyu wang <
>> [EMAIL PROTECTED]>
>> > > >> wrote:
>> > > >> > Hi all,
>> > > >> >
>> > > >> > I am running kafka-0.7.0 and found that even though expired
>> segments
>> > > are
>> > > >> > deleted by broker, the disk space is not released. So, du shows
>> > that a
>> > > >> > partition is 450 GB, but the total size of the files in the
>> > partition
>> > > is
>> > > >> > only about 250GB. Is this a bug? We are using raid10.
>> > > >> >
>> > > >> > Thanks,
>> > > >> >
>> > > >> > -Xiaoyu
>> > > >>
>> > > >
>> > > >
>> > >
>> >
>>
>
>

There is no preallocation in Kafka. I do remember seeing incorrect results
from du before.

Thanks,

Jun

On Thu, Aug 9, 2012 at 10:29 AM, xiaoyu wang <[EMAIL PROTECTED]> wrote:

> Does kafka pre-allocate 1GB for each segment? It shows that these segment
> files are all open and used by kafka broker.
>
> On Wed, Aug 8, 2012 at 4:14 PM, xiaoyu wang <[EMAIL PROTECTED]> wrote:
>
> > Jun,
> >
> > this does not seem to be a du problem because after restarting brokers,
> du
> > and ls show the same file size.
> >
> >
> > On Wed, Aug 8, 2012 at 3:55 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> >> Hmm, this seems like an issue with du.
> >>
> >> Thanks
> >>
> >> Jun
> >>
> >> On Wed, Aug 8, 2012 at 3:17 PM, xiaoyu wang <[EMAIL PROTECTED]>
> >> wrote:
> >>
> >> > Here is a sample. The highlighted lines are those showing different
> >> sizes
> >> > in du and ls
> >> >
> >> > *du output
> >> > *
> >> > 513M    ./00000001197773741184.kafka
> >> > 1.0G    ./00000001200458096891.kafka
> >> > 1.0G    ./00000001200994968784.kafka
> >> > 1.0G    ./00000001201531839884.kafka
> >> > 1.0G    ./00000001202068711094.kafka
> >> > 513M    ./00000001011479389046.kafka
> >> >
> >> > *ls output
> >> > *
> >> > -rw-r--r--  1 kafka kafka 536871053 Aug  8 13:01
> >> 00000001197773741184.kafka
> >> > -rw-r--r--  1 kafka kafka 536871893 Aug  8 13:36
> >> 00000001200458096891.kafka
> >> > -rw-r--r--  1 kafka kafka 536871100 Aug  8 13:45
> >> 00000001200994968784.kafka
> >> > -rw-r--r--  1 kafka kafka 536871210 Aug  8 13:49
> >> 00000001201531839884.kafka
> >> > -rw-r--r--  1 kafka kafka 536871063 Aug  8 14:00
> >> 00000001202068711094.kafka
> >> > -rw-r--r--  1 kafka kafka 536871139 Aug  6 07:02
> >> 00000001011479389046.kafka
> >> >
> >> > On Wed, Aug 8, 2012 at 2:58 PM, Joel Koshy <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> > > The mailing list does not accept attachments - you will need to
> either
> >> > > paste here or externally, say on gist.
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Joel
> >> > >
> >> > > On Wed, Aug 8, 2012 at 2:52 PM, xiaoyu wang <[EMAIL PROTECTED]>
> >> > wrote:
> >> > >
> >> > > > Neha,
> >> > > >
> >> > > > I have attached the du and ls output for one of the partitions.
> >> Looks
> >> > > like
> >> > > > for some segments, du is showing 1GB while ls shows 513MB. Let me
> >> know
> >> > if
> >> > > > you need additional info.
> >> > > >
> >> > > > Thanks,
> >> > > >
> >> > > > -Xiaoyu
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Wed, Aug 8, 2012 at 12:29 PM, Neha Narkhede <
> >> > [EMAIL PROTECTED]
> >> > > >wrote:
> >> > > >
> >> > > >> Xiaoyu,
> >> > > >>
> >> > > >> Please could you paste the log statements that specify which
> files
> >> are
> >> > > >> deleted by the broker and also the detailed output by du on the
> >> Kafka
> >> > > >> log directory ?
> >> > > >> If the broker says it has garbage collected some log segments,
> >> space
> >> > > >> should free up.
> >> > > >>
> >> > > >> Thanks,
> >> > > >> Neha
> >> > > >>
> >> > > >> On Wed, Aug 8, 2012 at 12:22 PM, xiaoyu wang <
> >> [EMAIL PROTECTED]>
> >> > > >> wrote:
> >> > > >> > Hi all,
> >> > > >> >
> >> > > >> > I am running kafka-0.7.0 and found that even though expired
> >> segments
> >> > > are
> >> > > >> > deleted by broker, the disk space is not released. So, du shows
> >> > that a
> >> > > >> > partition is 450 GB, but the total size of the files in the
> >> > partition
> >> > > is
> >> > > >> > only about 250GB. Is this a bug? We are using raid10.
> >> > > >> >
> >> > > >> > Thanks,
> >> > > >> >
> >> > > >> > -Xiaoyu
> >> > > >>
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>