-
Re: Closing .chukwa files in collector at regular fixed offset interval to avoid time interval driftBill Graham 2011-06-01, 00:16
I work with Shweta and this is something we've been considering for a while.
The idea is that collectors could be configured to close their files at some
fixed offset after a the end of the interval. In our case, we'd like files
to close at 30 seconds after the end of a 5 minute period. This will help us
with a couple of edge cases that the current approach causes for us.
Backward compatibility and default behavior would of course be maintained.
On Tue, May 31, 2011 at 2:12 PM, Eric Yang <[EMAIL PROTECTED]> wrote:
> +1 for JIRA, nice to have feature.
>
> Regards,
> Eric
>
> On 5/31/11 2:01 PM, "Ariel Rabkin" <[EMAIL PROTECTED]> wrote:
>
> Also.
>
> This sounds like something fairly site-specific. It would be really
> good to have unit tests so we don't break this
> feature in the future.
>
> --Ari
>
> On Tue, May 31, 2011 at 2:01 PM, Ariel Rabkin <[EMAIL PROTECTED]> wrote:
> > Not quite sure I understand the use case, but if it seems useful,
> > sure, open a JIRA.
> >
> > --Ari
> >
> > On Tue, May 31, 2011 at 11:33 AM, Shweta Shah <[EMAIL PROTECTED]>
> wrote:
> >> Hi,
> >>
> >> We have a system built on top of Chukwa that introduces the need to have
> all
> >> sequence files for a given rotateInterval closed and available at a
> >> predictable time. Currently, we are experiencing some drift in the
> times
> >> that the sequence files are closed due to the way the TimerTask is
> scheduled
> >> in the SeqFileWriter class. We would like to submit a solution that
> will
> >> allow people to configure the time all collectors should close their
> files
> >> for processing in a given interval, while still supporting the default
> >> functionality. We have written the code and are currently testing the
> >> functionality, but would like to know if anyone has any feedback on this
> >> development before submitting a JIRA ticket.
> >>
> >> - Shweta
> >>
> >
> >
> >
> > --
> > Ari Rabkin [EMAIL PROTECTED]
> > UC Berkeley Computer Science Department
> >
>
>
>
> --
> Ari Rabkin [EMAIL PROTECTED]
> UC Berkeley Computer Science Department
>
>
The idea is that collectors could be configured to close their files at some
fixed offset after a the end of the interval. In our case, we'd like files
to close at 30 seconds after the end of a 5 minute period. This will help us
with a couple of edge cases that the current approach causes for us.
Backward compatibility and default behavior would of course be maintained.
On Tue, May 31, 2011 at 2:12 PM, Eric Yang <[EMAIL PROTECTED]> wrote:
> +1 for JIRA, nice to have feature.
>
> Regards,
> Eric
>
> On 5/31/11 2:01 PM, "Ariel Rabkin" <[EMAIL PROTECTED]> wrote:
>
> Also.
>
> This sounds like something fairly site-specific. It would be really
> good to have unit tests so we don't break this
> feature in the future.
>
> --Ari
>
> On Tue, May 31, 2011 at 2:01 PM, Ariel Rabkin <[EMAIL PROTECTED]> wrote:
> > Not quite sure I understand the use case, but if it seems useful,
> > sure, open a JIRA.
> >
> > --Ari
> >
> > On Tue, May 31, 2011 at 11:33 AM, Shweta Shah <[EMAIL PROTECTED]>
> wrote:
> >> Hi,
> >>
> >> We have a system built on top of Chukwa that introduces the need to have
> all
> >> sequence files for a given rotateInterval closed and available at a
> >> predictable time. Currently, we are experiencing some drift in the
> times
> >> that the sequence files are closed due to the way the TimerTask is
> scheduled
> >> in the SeqFileWriter class. We would like to submit a solution that
> will
> >> allow people to configure the time all collectors should close their
> files
> >> for processing in a given interval, while still supporting the default
> >> functionality. We have written the code and are currently testing the
> >> functionality, but would like to know if anyone has any feedback on this
> >> development before submitting a JIRA ticket.
> >>
> >> - Shweta
> >>
> >
> >
> >
> > --
> > Ari Rabkin [EMAIL PROTECTED]
> > UC Berkeley Computer Science Department
> >
>
>
>
> --
> Ari Rabkin [EMAIL PROTECTED]
> UC Berkeley Computer Science Department
>
>
