Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # dev >> Closing .chukwa files in collector at regular fixed offset interval to avoid time interval drift


Copy link to this message
-
Re: Closing .chukwa files in collector at regular fixed offset interval to avoid time interval drift
I work with Shweta and this is something we've been considering for a while.
The idea is that collectors could be configured to close their files at some
fixed offset after a the end of the interval. In our case, we'd like files
to close at 30 seconds after the end of a 5 minute period. This will help us
with a couple of edge cases that the current approach causes for us.
Backward compatibility and default behavior would of course be maintained.
On Tue, May 31, 2011 at 2:12 PM, Eric Yang <[EMAIL PROTECTED]> wrote:

> +1 for JIRA, nice to have feature.
>
> Regards,
> Eric
>
> On 5/31/11 2:01 PM, "Ariel Rabkin" <[EMAIL PROTECTED]> wrote:
>
> Also.
>
> This sounds like something fairly site-specific. It would be really
> good to have unit tests so we don't break this
> feature in the future.
>
> --Ari
>
> On Tue, May 31, 2011 at 2:01 PM, Ariel Rabkin <[EMAIL PROTECTED]> wrote:
> > Not quite sure I understand the use case, but if it seems useful,
> > sure, open a JIRA.
> >
> > --Ari
> >
> > On Tue, May 31, 2011 at 11:33 AM, Shweta Shah <[EMAIL PROTECTED]>
> wrote:
> >> Hi,
> >>
> >> We have a system built on top of Chukwa that introduces the need to have
> all
> >> sequence files for a given rotateInterval closed and available at a
> >> predictable time.  Currently, we are experiencing some drift in the
> times
> >> that the sequence files are closed due to the way the TimerTask is
> scheduled
> >> in the SeqFileWriter class.  We would like to submit a solution that
> will
> >> allow people to configure the time all collectors should close their
> files
> >> for processing in a given interval, while still supporting the default
> >> functionality.  We have written the code and are currently testing the
> >> functionality, but would like to know if anyone has any feedback on this
> >> development before submitting a JIRA ticket.
> >>
> >> - Shweta
> >>
> >
> >
> >
> > --
> > Ari Rabkin [EMAIL PROTECTED]
> > UC Berkeley Computer Science Department
> >
>
>
>
> --
> Ari Rabkin [EMAIL PROTECTED]
> UC Berkeley Computer Science Department
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB