Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> lzo file rolling on Cloudera Flume


+
Devendra Tagare 2013-09-25, 09:51
+
Edwin Chiu 2013-09-25, 16:22
Copy link to this message
-
Re: lzo file rolling on Cloudera Flume
Can you try setting hdfs.idleTimeout to a small value (much smaller than roll timeout) and then see if the file gets rolled correctly?
Thanks,
Hari
On Wednesday, September 25, 2013 at 9:22 AM, Edwin Chiu wrote:

> Hi,
>
> I have also observed this behavior with gzip and bz2. I'm using flume-1.3.1.1.3.0.0-107.el6.noarch under Hortonworks 1.3.
>
> Best,
>
>
> - Edwin
>
> On Wed, Sep 25, 2013 at 2:51 AM, Devendra Tagare <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > Hi,
> >
> > I am using Cloudera Flume (apache-flume-1.3.0-cdh4.3.1)
> >
> > I have observed a large number of *.lzo.tmp files not getting rolled even though their file sizes are non-zero and the roll time has elapsed
> >
> > Regards,
> > Dev

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB