Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> HDFS sink leaves .tmp files


Copy link to this message
-
Re: HDFS sink leaves .tmp files
Yeah - that's built against CDH3 though. But I think it would work anyway.
On Mon, Sep 10, 2012 at 1:08 PM, Bhaskar V. Karambelkar <[EMAIL PROTECTED]
> wrote:

> Don't know about RPM, but there's a 1.2.x tarball of the 1.2 build @
> http://archive.cloudera.com/cdh/3/flume-ng-1.2.0-cdh3u5.tar.gz
>
>
> On Mon, Sep 10, 2012 at 3:01 PM, Chris Neal <[EMAIL PROTECTED]> wrote:
>
>> Just checked, and from Cloudera, 1.1.0+121-1.cdh4.0.1.p0.1.el6 is still
>> the latest from their yum repo.
>>
>>
>> On Mon, Sep 10, 2012 at 1:59 PM, Chris Neal <[EMAIL PROTECTED]> wrote:
>>
>>> I'm using a combination :)
>>>
>>> The application tier is 1.3.0-SNAPSHOT
>>> The HDFS tier is CentOS, and I grabbed the latest (at the time) from the
>>> CDH repo.  It's version is:  1.1.0+121-1.cdh4.0.1.p0.1.el6
>>>
>>> If the issue is on the HDFS sink side, that it could definitely be in my
>>> version!
>>> I'll check if Cloudera has a more recent version to update to.
>>>
>>> Thanks!
>>>  Chris
>>>
>>>
>>> On Mon, Sep 10, 2012 at 12:37 PM, Kathleen Ting <[EMAIL PROTECTED]>wrote:
>>>
>>>> Chris, Eran, this appears to be FLUME-1238, which was fixed in
>>>> Flume-1.2.0. Can you let me know if you are using Flume-1.2.0?
>>>>
>>>> Thanks, Kathleen
>>>>
>>>> On Mon, Sep 10, 2012 at 8:21 AM, Chris Neal <[EMAIL PROTECTED]> wrote:
>>>> > Glad to know it's not just me :)
>>>> >
>>>> >
>>>> > On Mon, Sep 10, 2012 at 10:16 AM, Eran Kutner <[EMAIL PROTECTED]> wrote:
>>>> >>
>>>> >> I have the same problem. I roll every 1 minute so I have tons of
>>>> those
>>>> >> .tmp files.
>>>> >>
>>>> >> -eran
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Mon, Sep 10, 2012 at 6:02 PM, Chris Neal <[EMAIL PROTECTED]>
>>>> wrote:
>>>> >>>
>>>> >>> I'm still seeing this consistently every 24 hour period.  Does this
>>>> sound
>>>> >>> like a configuration issue, an issue with the Exec source, or an
>>>> issue with
>>>> >>> the HDFS sink?
>>>> >>>
>>>> >>> Thanks!
>>>> >>>
>>>> >>>
>>>> >>> On Wed, Aug 29, 2012 at 9:18 AM, Chris Neal <[EMAIL PROTECTED]>
>>>> wrote:
>>>> >>>>
>>>> >>>> Hi all,
>>>> >>>>
>>>> >>>> I have an Exec Source running a tail -F on a log4J-generated log
>>>> file
>>>> >>>> that gets rolled once a day.  It seems that when log4J rolls the
>>>> file to the
>>>> >>>> new date, the hdfs sink ends up with a .tmp file.  I haven't
>>>> figured out if
>>>> >>>> there is any data loss yet, but was curious if this is expected
>>>> behavior?
>>>> >>>>
>>>> >>>> Thanks for your time.
>>>> >>>> Chris
>>>> >>>
>>>> >>>
>>>> >>
>>>> >
>>>>
>>>
>>>
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB