Juhani Connolly
2012-11-29, 05:20
Mohit Anchlia
2012-11-29, 16:46
Mohit Anchlia
2012-11-11, 16:32
Mike Percy
2012-11-15, 19:54
Mohit Anchlia
2012-11-16, 04:14
Mohit Anchlia
2012-11-16, 04:16
Juhani Connolly
2012-11-16, 07:23
Mike Percy
2012-11-20, 19:16
Mohit Anchlia
2012-11-20, 20:53
Brock Noland
2012-11-20, 20:56
Mike Percy
2012-11-20, 23:11
Mohit Anchlia
2012-11-20, 23:24
Alexander Alten-Lorenz
2012-11-16, 06:51
-
Re: .tmp in hdfs sinkJuhani Connolly 2012-11-29, 05:20
The changes are in both the 1.3 RC5 and in the 1.4 trunk
On 11/29/2012 01:26 PM, Mohit Anchlia wrote:
> If I grab the last snapshot would I get these changes?
>
> On Tue, Nov 20, 2012 at 3:24 PM, Mohit Anchlia <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
> that's awesome!
>
>
> On Tue, Nov 20, 2012 at 3:11 PM, Mike Percy <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
> Mohit,
> No problem, but Juhani did all the work. :)
>
> The behavior is that you can configure an HDFS sink to close a
> file if it hasn't gotten any writes in some time. After it's
> been idle for 5 minutes or something, it gets closed. If you
> get a "late" event that goes to the same path after the file
> is closed, it will just create a new file in the same path as
> usual.
>
> Regards,
> Mike
>
>
> On Tue, Nov 20, 2012 at 12:56 PM, Brock Noland
> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote:
>
> We are currently voting on a 1.3.0 RC on the dev@ list:
>
> http://s.apache.org/OQ0W
>
> You don't have to be a committer to vote! :)
>
> Brock
>
> On Tue, Nov 20, 2012 at 2:53 PM, Mohit Anchlia
> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> wrote:
> > Thanks a lot!! Now with this what should be the expected
> behaviour? After
> > file is closed a new file is created for writes that
> come after closing the
> > file?
> >
> > Thanks again for committing this change. Do you know
> when 1.3.0 is out? I am
> > currently using the snapshot version of 1.3.0
> >
> > On Tue, Nov 20, 2012 at 11:16 AM, Mike Percy
> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote:
> >>
> >> Mohit,
> >> FLUME-1660 is now committed and it will be in 1.3.0. In
> the case where you
> >> are using 1.2.0, I suggest running with
> hdfs.rollInterval set so the files
> >> will roll normally.
> >>
> >> Regards,
> >> Mike
> >>
> >>
> >> On Thu, Nov 15, 2012 at 11:23 PM, Juhani Connolly
> >> <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
> >>>
> >>> I am actually working on a patch for exactly this,
> refer to FLUME-1660
> >>>
> >>> The patch is on review board right now, I fixed a
> corner case issue that
> >>> came up with unit testing, but the implementation is
> not really to my
> >>> satisfaction. If you are interested please have a look
> and add your opinion.
> >>>
> >>> https://issues.apache.org/jira/browse/FLUME-1660
> >>> https://reviews.apache.org/r/7659/
> >>>
> >>>
> >>> On 11/16/2012 01:16 PM, Mohit Anchlia wrote:
> >>>
> >>> Another question I had was about rollover. What's the
> best way to
> >>> rollover files in reasonable timeframe? For instance
> our path is YY/MM/DD/HH
> >>> so every hour there is new file and the -1 hr is just
> sitting with .tmp and
> >>> it takes sometimes even hour before .tmp is closed and
> renamed to .snappy.
> >>> In this situation is there a way to tell flume to
> rollover files sooner
> >>> based on some idle time limit?
> >>>
> >>> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia
> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> >>> wrote:
> >>>>
> >>>> Thanks Mike it makes sense. Anyway I can help?
On 11/29/2012 01:26 PM, Mohit Anchlia wrote:
> If I grab the last snapshot would I get these changes?
>
> On Tue, Nov 20, 2012 at 3:24 PM, Mohit Anchlia <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
> that's awesome!
>
>
> On Tue, Nov 20, 2012 at 3:11 PM, Mike Percy <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
> Mohit,
> No problem, but Juhani did all the work. :)
>
> The behavior is that you can configure an HDFS sink to close a
> file if it hasn't gotten any writes in some time. After it's
> been idle for 5 minutes or something, it gets closed. If you
> get a "late" event that goes to the same path after the file
> is closed, it will just create a new file in the same path as
> usual.
>
> Regards,
> Mike
>
>
> On Tue, Nov 20, 2012 at 12:56 PM, Brock Noland
> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote:
>
> We are currently voting on a 1.3.0 RC on the dev@ list:
>
> http://s.apache.org/OQ0W
>
> You don't have to be a committer to vote! :)
>
> Brock
>
> On Tue, Nov 20, 2012 at 2:53 PM, Mohit Anchlia
> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> wrote:
> > Thanks a lot!! Now with this what should be the expected
> behaviour? After
> > file is closed a new file is created for writes that
> come after closing the
> > file?
> >
> > Thanks again for committing this change. Do you know
> when 1.3.0 is out? I am
> > currently using the snapshot version of 1.3.0
> >
> > On Tue, Nov 20, 2012 at 11:16 AM, Mike Percy
> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote:
> >>
> >> Mohit,
> >> FLUME-1660 is now committed and it will be in 1.3.0. In
> the case where you
> >> are using 1.2.0, I suggest running with
> hdfs.rollInterval set so the files
> >> will roll normally.
> >>
> >> Regards,
> >> Mike
> >>
> >>
> >> On Thu, Nov 15, 2012 at 11:23 PM, Juhani Connolly
> >> <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
> >>>
> >>> I am actually working on a patch for exactly this,
> refer to FLUME-1660
> >>>
> >>> The patch is on review board right now, I fixed a
> corner case issue that
> >>> came up with unit testing, but the implementation is
> not really to my
> >>> satisfaction. If you are interested please have a look
> and add your opinion.
> >>>
> >>> https://issues.apache.org/jira/browse/FLUME-1660
> >>> https://reviews.apache.org/r/7659/
> >>>
> >>>
> >>> On 11/16/2012 01:16 PM, Mohit Anchlia wrote:
> >>>
> >>> Another question I had was about rollover. What's the
> best way to
> >>> rollover files in reasonable timeframe? For instance
> our path is YY/MM/DD/HH
> >>> so every hour there is new file and the -1 hr is just
> sitting with .tmp and
> >>> it takes sometimes even hour before .tmp is closed and
> renamed to .snappy.
> >>> In this situation is there a way to tell flume to
> rollover files sooner
> >>> based on some idle time limit?
> >>>
> >>> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia
> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> >>> wrote:
> >>>>
> >>>> Thanks Mike it makes sense. Anyway I can help?
+
Juhani Connolly 2012-11-29, 05:20
-
Re: .tmp in hdfs sinkMohit Anchlia 2012-11-29, 16:46
Thanks for your response so far. I checkedout flume-1.3.0 and have built
it. My next question is the property hdfs.closeIdleTimeout correct? Do I
need to set any other property? My current config looks like and I write by
YYYY/MM/DD/HH format so essentially I get 1-2 files per hour.
webanalytics.sinks.hdfsSink.hdfs.filePrefix = web
webanalytics.sinks.hdfsSink.hdfs.rollInterval = 4000
webanalytics.sinks.hdfsSink.hdfs.rollCount = 20000000
#webanalytics.sinks.hdfsSink.hdfs.rollCount = 40000
webanalytics.sinks.hdfsSink.hdfs.rollSize = 15000000000
webanalytics.sinks.hdfsSink.hdfs.fileType = SequenceFile
webanalytics.sinks.hdfsSink.hdfs.writeFormat = Text
webanalytics.sinks.hdfsSink.hdfs.codeC = snappy
On Wed, Nov 28, 2012 at 9:20 PM, Juhani Connolly <
[EMAIL PROTECTED]> wrote:
> The changes are in both the 1.3 RC5 and in the 1.4 trunk
>
>
> On 11/29/2012 01:26 PM, Mohit Anchlia wrote:
>
> If I grab the last snapshot would I get these changes?
>
> On Tue, Nov 20, 2012 at 3:24 PM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>
>> that's awesome!
>>
>>
>> On Tue, Nov 20, 2012 at 3:11 PM, Mike Percy <[EMAIL PROTECTED]> wrote:
>>
>>> Mohit,
>>> No problem, but Juhani did all the work. :)
>>>
>>> The behavior is that you can configure an HDFS sink to close a file if
>>> it hasn't gotten any writes in some time. After it's been idle for 5
>>> minutes or something, it gets closed. If you get a "late" event that goes
>>> to the same path after the file is closed, it will just create a new file
>>> in the same path as usual.
>>>
>>> Regards,
>>> Mike
>>>
>>>
>>> On Tue, Nov 20, 2012 at 12:56 PM, Brock Noland <[EMAIL PROTECTED]>wrote:
>>>
>>>> We are currently voting on a 1.3.0 RC on the dev@ list:
>>>>
>>>> http://s.apache.org/OQ0W
>>>>
>>>> You don't have to be a committer to vote! :)
>>>>
>>>> Brock
>>>>
>>>> On Tue, Nov 20, 2012 at 2:53 PM, Mohit Anchlia <[EMAIL PROTECTED]>
>>>> wrote:
>>>> > Thanks a lot!! Now with this what should be the expected behaviour?
>>>> After
>>>> > file is closed a new file is created for writes that come after
>>>> closing the
>>>> > file?
>>>> >
>>>> > Thanks again for committing this change. Do you know when 1.3.0 is
>>>> out? I am
>>>> > currently using the snapshot version of 1.3.0
>>>> >
>>>> > On Tue, Nov 20, 2012 at 11:16 AM, Mike Percy <[EMAIL PROTECTED]>
>>>> wrote:
>>>> >>
>>>> >> Mohit,
>>>> >> FLUME-1660 is now committed and it will be in 1.3.0. In the case
>>>> where you
>>>> >> are using 1.2.0, I suggest running with hdfs.rollInterval set so the
>>>> files
>>>> >> will roll normally.
>>>> >>
>>>> >> Regards,
>>>> >> Mike
>>>> >>
>>>> >>
>>>> >> On Thu, Nov 15, 2012 at 11:23 PM, Juhani Connolly
>>>> >> <[EMAIL PROTECTED]> wrote:
>>>> >>>
>>>> >>> I am actually working on a patch for exactly this, refer to
>>>> FLUME-1660
>>>> >>>
>>>> >>> The patch is on review board right now, I fixed a corner case issue
>>>> that
>>>> >>> came up with unit testing, but the implementation is not really to
>>>> my
>>>> >>> satisfaction. If you are interested please have a look and add your
>>>> opinion.
>>>> >>>
>>>> >>> https://issues.apache.org/jira/browse/FLUME-1660
>>>> >>> https://reviews.apache.org/r/7659/
>>>> >>>
>>>> >>>
>>>> >>> On 11/16/2012 01:16 PM, Mohit Anchlia wrote:
>>>> >>>
>>>> >>> Another question I had was about rollover. What's the best way to
>>>> >>> rollover files in reasonable timeframe? For instance our path is
>>>> YY/MM/DD/HH
>>>> >>> so every hour there is new file and the -1 hr is just sitting with
>>>> .tmp and
>>>> >>> it takes sometimes even hour before .tmp is closed and renamed to
>>>> .snappy.
>>>> >>> In this situation is there a way to tell flume to rollover files
>>>> sooner
>>>> >>> based on some idle time limit?
>>>> >>>
>>>> >>> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <
>>>> [EMAIL PROTECTED]>
>>>> >>> wrote:
>>>> >>>>
>>>> >>>> Thanks Mike it makes sense. Anyway I can help?
>>>> >>>>
>>>> >>>>
>>>> >>>> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]>
it. My next question is the property hdfs.closeIdleTimeout correct? Do I
need to set any other property? My current config looks like and I write by
YYYY/MM/DD/HH format so essentially I get 1-2 files per hour.
webanalytics.sinks.hdfsSink.hdfs.filePrefix = web
webanalytics.sinks.hdfsSink.hdfs.rollInterval = 4000
webanalytics.sinks.hdfsSink.hdfs.rollCount = 20000000
#webanalytics.sinks.hdfsSink.hdfs.rollCount = 40000
webanalytics.sinks.hdfsSink.hdfs.rollSize = 15000000000
webanalytics.sinks.hdfsSink.hdfs.fileType = SequenceFile
webanalytics.sinks.hdfsSink.hdfs.writeFormat = Text
webanalytics.sinks.hdfsSink.hdfs.codeC = snappy
On Wed, Nov 28, 2012 at 9:20 PM, Juhani Connolly <
[EMAIL PROTECTED]> wrote:
> The changes are in both the 1.3 RC5 and in the 1.4 trunk
>
>
> On 11/29/2012 01:26 PM, Mohit Anchlia wrote:
>
> If I grab the last snapshot would I get these changes?
>
> On Tue, Nov 20, 2012 at 3:24 PM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>
>> that's awesome!
>>
>>
>> On Tue, Nov 20, 2012 at 3:11 PM, Mike Percy <[EMAIL PROTECTED]> wrote:
>>
>>> Mohit,
>>> No problem, but Juhani did all the work. :)
>>>
>>> The behavior is that you can configure an HDFS sink to close a file if
>>> it hasn't gotten any writes in some time. After it's been idle for 5
>>> minutes or something, it gets closed. If you get a "late" event that goes
>>> to the same path after the file is closed, it will just create a new file
>>> in the same path as usual.
>>>
>>> Regards,
>>> Mike
>>>
>>>
>>> On Tue, Nov 20, 2012 at 12:56 PM, Brock Noland <[EMAIL PROTECTED]>wrote:
>>>
>>>> We are currently voting on a 1.3.0 RC on the dev@ list:
>>>>
>>>> http://s.apache.org/OQ0W
>>>>
>>>> You don't have to be a committer to vote! :)
>>>>
>>>> Brock
>>>>
>>>> On Tue, Nov 20, 2012 at 2:53 PM, Mohit Anchlia <[EMAIL PROTECTED]>
>>>> wrote:
>>>> > Thanks a lot!! Now with this what should be the expected behaviour?
>>>> After
>>>> > file is closed a new file is created for writes that come after
>>>> closing the
>>>> > file?
>>>> >
>>>> > Thanks again for committing this change. Do you know when 1.3.0 is
>>>> out? I am
>>>> > currently using the snapshot version of 1.3.0
>>>> >
>>>> > On Tue, Nov 20, 2012 at 11:16 AM, Mike Percy <[EMAIL PROTECTED]>
>>>> wrote:
>>>> >>
>>>> >> Mohit,
>>>> >> FLUME-1660 is now committed and it will be in 1.3.0. In the case
>>>> where you
>>>> >> are using 1.2.0, I suggest running with hdfs.rollInterval set so the
>>>> files
>>>> >> will roll normally.
>>>> >>
>>>> >> Regards,
>>>> >> Mike
>>>> >>
>>>> >>
>>>> >> On Thu, Nov 15, 2012 at 11:23 PM, Juhani Connolly
>>>> >> <[EMAIL PROTECTED]> wrote:
>>>> >>>
>>>> >>> I am actually working on a patch for exactly this, refer to
>>>> FLUME-1660
>>>> >>>
>>>> >>> The patch is on review board right now, I fixed a corner case issue
>>>> that
>>>> >>> came up with unit testing, but the implementation is not really to
>>>> my
>>>> >>> satisfaction. If you are interested please have a look and add your
>>>> opinion.
>>>> >>>
>>>> >>> https://issues.apache.org/jira/browse/FLUME-1660
>>>> >>> https://reviews.apache.org/r/7659/
>>>> >>>
>>>> >>>
>>>> >>> On 11/16/2012 01:16 PM, Mohit Anchlia wrote:
>>>> >>>
>>>> >>> Another question I had was about rollover. What's the best way to
>>>> >>> rollover files in reasonable timeframe? For instance our path is
>>>> YY/MM/DD/HH
>>>> >>> so every hour there is new file and the -1 hr is just sitting with
>>>> .tmp and
>>>> >>> it takes sometimes even hour before .tmp is closed and renamed to
>>>> .snappy.
>>>> >>> In this situation is there a way to tell flume to rollover files
>>>> sooner
>>>> >>> based on some idle time limit?
>>>> >>>
>>>> >>> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <
>>>> [EMAIL PROTECTED]>
>>>> >>> wrote:
>>>> >>>>
>>>> >>>> Thanks Mike it makes sense. Anyway I can help?
>>>> >>>>
>>>> >>>>
>>>> >>>> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]>
+
Mohit Anchlia 2012-11-29, 16:46
-
.tmp in hdfs sinkMohit Anchlia 2012-11-11, 16:32
What we are seeing is that if flume gets killed either because of server
failure or other reasons, it keeps around the .tmp file. Sometimes for
whatever reasons .tmp file is not readable. Is there a way to rollover .tmp
file more gracefully?
failure or other reasons, it keeps around the .tmp file. Sometimes for
whatever reasons .tmp file is not readable. Is there a way to rollover .tmp
file more gracefully?
+
Mohit Anchlia 2012-11-11, 16:32
-
Re: .tmp in hdfs sinkMike Percy 2012-11-15, 19:54
Hi Mohit, this is a complicated issue. I've filed
https://issues.apache.org/jira/browse/FLUME-1714 to track it.
In short, it would require a non-trivial amount of work to implement this,
and it would need to be done carefully. I agree that it would be better if
Flume handled this case more gracefully than it does today. Today, Flume
assumes that you have some job that would go and clean up the .tmp files as
needed, and that you understand that they could be partially written if a
crash occurred.
Regards,
Mike
On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
> What we are seeing is that if flume gets killed either because of server
> failure or other reasons, it keeps around the .tmp file. Sometimes for
> whatever reasons .tmp file is not readable. Is there a way to rollover .tmp
> file more gracefully?
>
https://issues.apache.org/jira/browse/FLUME-1714 to track it.
In short, it would require a non-trivial amount of work to implement this,
and it would need to be done carefully. I agree that it would be better if
Flume handled this case more gracefully than it does today. Today, Flume
assumes that you have some job that would go and clean up the .tmp files as
needed, and that you understand that they could be partially written if a
crash occurred.
Regards,
Mike
On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
> What we are seeing is that if flume gets killed either because of server
> failure or other reasons, it keeps around the .tmp file. Sometimes for
> whatever reasons .tmp file is not readable. Is there a way to rollover .tmp
> file more gracefully?
>
+
Mike Percy 2012-11-15, 19:54
-
Re: .tmp in hdfs sinkMohit Anchlia 2012-11-16, 04:14
Thanks Mike it makes sense. Anyway I can help?
On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
> Hi Mohit, this is a complicated issue. I've filed
> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
>
> In short, it would require a non-trivial amount of work to implement this,
> and it would need to be done carefully. I agree that it would be better if
> Flume handled this case more gracefully than it does today. Today, Flume
> assumes that you have some job that would go and clean up the .tmp files as
> needed, and that you understand that they could be partially written if a
> crash occurred.
>
> Regards,
> Mike
>
>
> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>
>> What we are seeing is that if flume gets killed either because of server
>> failure or other reasons, it keeps around the .tmp file. Sometimes for
>> whatever reasons .tmp file is not readable. Is there a way to rollover .tmp
>> file more gracefully?
>>
>
>
On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
> Hi Mohit, this is a complicated issue. I've filed
> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
>
> In short, it would require a non-trivial amount of work to implement this,
> and it would need to be done carefully. I agree that it would be better if
> Flume handled this case more gracefully than it does today. Today, Flume
> assumes that you have some job that would go and clean up the .tmp files as
> needed, and that you understand that they could be partially written if a
> crash occurred.
>
> Regards,
> Mike
>
>
> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>
>> What we are seeing is that if flume gets killed either because of server
>> failure or other reasons, it keeps around the .tmp file. Sometimes for
>> whatever reasons .tmp file is not readable. Is there a way to rollover .tmp
>> file more gracefully?
>>
>
>
+
Mohit Anchlia 2012-11-16, 04:14
-
Re: .tmp in hdfs sinkMohit Anchlia 2012-11-16, 04:16
Another question I had was about rollover. What's the best way to rollover
files in reasonable timeframe? For instance our path is YY/MM/DD/HH so
every hour there is new file and the -1 hr is just sitting with .tmp and it
takes sometimes even hour before .tmp is closed and renamed to .snappy. In
this situation is there a way to tell flume to rollover files sooner based
on some idle time limit?
On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
> Thanks Mike it makes sense. Anyway I can help?
>
>
> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>
>> Hi Mohit, this is a complicated issue. I've filed
>> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
>>
>> In short, it would require a non-trivial amount of work to implement
>> this, and it would need to be done carefully. I agree that it would be
>> better if Flume handled this case more gracefully than it does today.
>> Today, Flume assumes that you have some job that would go and clean up the
>> .tmp files as needed, and that you understand that they could be partially
>> written if a crash occurred.
>>
>> Regards,
>> Mike
>>
>>
>> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>>
>>> What we are seeing is that if flume gets killed either because of server
>>> failure or other reasons, it keeps around the .tmp file. Sometimes for
>>> whatever reasons .tmp file is not readable. Is there a way to rollover .tmp
>>> file more gracefully?
>>>
>>
>>
>
files in reasonable timeframe? For instance our path is YY/MM/DD/HH so
every hour there is new file and the -1 hr is just sitting with .tmp and it
takes sometimes even hour before .tmp is closed and renamed to .snappy. In
this situation is there a way to tell flume to rollover files sooner based
on some idle time limit?
On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
> Thanks Mike it makes sense. Anyway I can help?
>
>
> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>
>> Hi Mohit, this is a complicated issue. I've filed
>> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
>>
>> In short, it would require a non-trivial amount of work to implement
>> this, and it would need to be done carefully. I agree that it would be
>> better if Flume handled this case more gracefully than it does today.
>> Today, Flume assumes that you have some job that would go and clean up the
>> .tmp files as needed, and that you understand that they could be partially
>> written if a crash occurred.
>>
>> Regards,
>> Mike
>>
>>
>> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>>
>>> What we are seeing is that if flume gets killed either because of server
>>> failure or other reasons, it keeps around the .tmp file. Sometimes for
>>> whatever reasons .tmp file is not readable. Is there a way to rollover .tmp
>>> file more gracefully?
>>>
>>
>>
>
+
Mohit Anchlia 2012-11-16, 04:16
-
Re: .tmp in hdfs sinkJuhani Connolly 2012-11-16, 07:23
I am actually working on a patch for exactly this, refer to FLUME-1660
The patch is on review board right now, I fixed a corner case issue that
came up with unit testing, but the implementation is not really to my
satisfaction. If you are interested please have a look and add your opinion.
https://issues.apache.org/jira/browse/FLUME-1660
https://reviews.apache.org/r/7659/
On 11/16/2012 01:16 PM, Mohit Anchlia wrote:
> Another question I had was about rollover. What's the best way to
> rollover files in reasonable timeframe? For instance our path is
> YY/MM/DD/HH so every hour there is new file and the -1 hr is just
> sitting with .tmp and it takes sometimes even hour before .tmp is
> closed and renamed to .snappy. In this situation is there a way to
> tell flume to rollover files sooner based on some idle time limit?
>
> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
> Thanks Mike it makes sense. Anyway I can help?
>
>
> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
> Hi Mohit, this is a complicated issue. I've filed
> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
>
> In short, it would require a non-trivial amount of work to
> implement this, and it would need to be done carefully. I
> agree that it would be better if Flume handled this case more
> gracefully than it does today. Today, Flume assumes that you
> have some job that would go and clean up the .tmp files as
> needed, and that you understand that they could be partially
> written if a crash occurred.
>
> Regards,
> Mike
>
>
> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia
> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote:
>
> What we are seeing is that if flume gets killed either
> because of server failure or other reasons, it keeps
> around the .tmp file. Sometimes for whatever reasons .tmp
> file is not readable. Is there a way to rollover .tmp file
> more gracefully?
>
>
>
>
The patch is on review board right now, I fixed a corner case issue that
came up with unit testing, but the implementation is not really to my
satisfaction. If you are interested please have a look and add your opinion.
https://issues.apache.org/jira/browse/FLUME-1660
https://reviews.apache.org/r/7659/
On 11/16/2012 01:16 PM, Mohit Anchlia wrote:
> Another question I had was about rollover. What's the best way to
> rollover files in reasonable timeframe? For instance our path is
> YY/MM/DD/HH so every hour there is new file and the -1 hr is just
> sitting with .tmp and it takes sometimes even hour before .tmp is
> closed and renamed to .snappy. In this situation is there a way to
> tell flume to rollover files sooner based on some idle time limit?
>
> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
> Thanks Mike it makes sense. Anyway I can help?
>
>
> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
> Hi Mohit, this is a complicated issue. I've filed
> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
>
> In short, it would require a non-trivial amount of work to
> implement this, and it would need to be done carefully. I
> agree that it would be better if Flume handled this case more
> gracefully than it does today. Today, Flume assumes that you
> have some job that would go and clean up the .tmp files as
> needed, and that you understand that they could be partially
> written if a crash occurred.
>
> Regards,
> Mike
>
>
> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia
> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote:
>
> What we are seeing is that if flume gets killed either
> because of server failure or other reasons, it keeps
> around the .tmp file. Sometimes for whatever reasons .tmp
> file is not readable. Is there a way to rollover .tmp file
> more gracefully?
>
>
>
>
+
Juhani Connolly 2012-11-16, 07:23
-
Re: .tmp in hdfs sinkMike Percy 2012-11-20, 19:16
Mohit,
FLUME-1660 is now committed and it will be in 1.3.0. In the case where you
are using 1.2.0, I suggest running with hdfs.rollInterval set so the files
will roll normally.
Regards,
Mike
On Thu, Nov 15, 2012 at 11:23 PM, Juhani Connolly <
[EMAIL PROTECTED]> wrote:
> I am actually working on a patch for exactly this, refer to FLUME-1660
>
> The patch is on review board right now, I fixed a corner case issue that
> came up with unit testing, but the implementation is not really to my
> satisfaction. If you are interested please have a look and add your opinion.
>
> https://issues.apache.org/jira/browse/FLUME-1660
> https://reviews.apache.org/r/7659/
>
>
> On 11/16/2012 01:16 PM, Mohit Anchlia wrote:
>
> Another question I had was about rollover. What's the best way to rollover
> files in reasonable timeframe? For instance our path is YY/MM/DD/HH so
> every hour there is new file and the -1 hr is just sitting with .tmp and it
> takes sometimes even hour before .tmp is closed and renamed to .snappy. In
> this situation is there a way to tell flume to rollover files sooner based
> on some idle time limit?
>
> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>
>> Thanks Mike it makes sense. Anyway I can help?
>>
>>
>> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>>
>>> Hi Mohit, this is a complicated issue. I've filed
>>> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
>>>
>>> In short, it would require a non-trivial amount of work to implement
>>> this, and it would need to be done carefully. I agree that it would be
>>> better if Flume handled this case more gracefully than it does today.
>>> Today, Flume assumes that you have some job that would go and clean up the
>>> .tmp files as needed, and that you understand that they could be partially
>>> written if a crash occurred.
>>>
>>> Regards,
>>> Mike
>>>
>>>
>>> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>>>
>>>> What we are seeing is that if flume gets killed either because of
>>>> server failure or other reasons, it keeps around the .tmp file. Sometimes
>>>> for whatever reasons .tmp file is not readable. Is there a way to rollover
>>>> .tmp file more gracefully?
>>>>
>>>
>>>
>>
>
>
FLUME-1660 is now committed and it will be in 1.3.0. In the case where you
are using 1.2.0, I suggest running with hdfs.rollInterval set so the files
will roll normally.
Regards,
Mike
On Thu, Nov 15, 2012 at 11:23 PM, Juhani Connolly <
[EMAIL PROTECTED]> wrote:
> I am actually working on a patch for exactly this, refer to FLUME-1660
>
> The patch is on review board right now, I fixed a corner case issue that
> came up with unit testing, but the implementation is not really to my
> satisfaction. If you are interested please have a look and add your opinion.
>
> https://issues.apache.org/jira/browse/FLUME-1660
> https://reviews.apache.org/r/7659/
>
>
> On 11/16/2012 01:16 PM, Mohit Anchlia wrote:
>
> Another question I had was about rollover. What's the best way to rollover
> files in reasonable timeframe? For instance our path is YY/MM/DD/HH so
> every hour there is new file and the -1 hr is just sitting with .tmp and it
> takes sometimes even hour before .tmp is closed and renamed to .snappy. In
> this situation is there a way to tell flume to rollover files sooner based
> on some idle time limit?
>
> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>
>> Thanks Mike it makes sense. Anyway I can help?
>>
>>
>> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>>
>>> Hi Mohit, this is a complicated issue. I've filed
>>> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
>>>
>>> In short, it would require a non-trivial amount of work to implement
>>> this, and it would need to be done carefully. I agree that it would be
>>> better if Flume handled this case more gracefully than it does today.
>>> Today, Flume assumes that you have some job that would go and clean up the
>>> .tmp files as needed, and that you understand that they could be partially
>>> written if a crash occurred.
>>>
>>> Regards,
>>> Mike
>>>
>>>
>>> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>>>
>>>> What we are seeing is that if flume gets killed either because of
>>>> server failure or other reasons, it keeps around the .tmp file. Sometimes
>>>> for whatever reasons .tmp file is not readable. Is there a way to rollover
>>>> .tmp file more gracefully?
>>>>
>>>
>>>
>>
>
>
+
Mike Percy 2012-11-20, 19:16
-
Re: .tmp in hdfs sinkMohit Anchlia 2012-11-20, 20:53
Thanks a lot!! Now with this what should be the expected behaviour? After
file is closed a new file is created for writes that come after closing the
file?
Thanks again for committing this change. Do you know when 1.3.0 is out? I
am currently using the snapshot version of 1.3.0
On Tue, Nov 20, 2012 at 11:16 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
> Mohit,
> FLUME-1660 is now committed and it will be in 1.3.0. In the case where you
> are using 1.2.0, I suggest running with hdfs.rollInterval set so the files
> will roll normally.
>
> Regards,
> Mike
>
>
> On Thu, Nov 15, 2012 at 11:23 PM, Juhani Connolly <
> [EMAIL PROTECTED]> wrote:
>
>> I am actually working on a patch for exactly this, refer to FLUME-1660
>>
>> The patch is on review board right now, I fixed a corner case issue that
>> came up with unit testing, but the implementation is not really to my
>> satisfaction. If you are interested please have a look and add your opinion.
>>
>> https://issues.apache.org/jira/browse/FLUME-1660
>> https://reviews.apache.org/r/7659/
>>
>>
>> On 11/16/2012 01:16 PM, Mohit Anchlia wrote:
>>
>> Another question I had was about rollover. What's the best way to
>> rollover files in reasonable timeframe? For instance our path is
>> YY/MM/DD/HH so every hour there is new file and the -1 hr is just sitting
>> with .tmp and it takes sometimes even hour before .tmp is closed and
>> renamed to .snappy. In this situation is there a way to tell flume to
>> rollover files sooner based on some idle time limit?
>>
>> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>>
>>> Thanks Mike it makes sense. Anyway I can help?
>>>
>>>
>>> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi Mohit, this is a complicated issue. I've filed
>>>> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
>>>>
>>>> In short, it would require a non-trivial amount of work to implement
>>>> this, and it would need to be done carefully. I agree that it would be
>>>> better if Flume handled this case more gracefully than it does today.
>>>> Today, Flume assumes that you have some job that would go and clean up the
>>>> .tmp files as needed, and that you understand that they could be partially
>>>> written if a crash occurred.
>>>>
>>>> Regards,
>>>> Mike
>>>>
>>>>
>>>> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> What we are seeing is that if flume gets killed either because of
>>>>> server failure or other reasons, it keeps around the .tmp file. Sometimes
>>>>> for whatever reasons .tmp file is not readable. Is there a way to rollover
>>>>> .tmp file more gracefully?
>>>>>
>>>>
>>>>
>>>
>>
>>
>
file is closed a new file is created for writes that come after closing the
file?
Thanks again for committing this change. Do you know when 1.3.0 is out? I
am currently using the snapshot version of 1.3.0
On Tue, Nov 20, 2012 at 11:16 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
> Mohit,
> FLUME-1660 is now committed and it will be in 1.3.0. In the case where you
> are using 1.2.0, I suggest running with hdfs.rollInterval set so the files
> will roll normally.
>
> Regards,
> Mike
>
>
> On Thu, Nov 15, 2012 at 11:23 PM, Juhani Connolly <
> [EMAIL PROTECTED]> wrote:
>
>> I am actually working on a patch for exactly this, refer to FLUME-1660
>>
>> The patch is on review board right now, I fixed a corner case issue that
>> came up with unit testing, but the implementation is not really to my
>> satisfaction. If you are interested please have a look and add your opinion.
>>
>> https://issues.apache.org/jira/browse/FLUME-1660
>> https://reviews.apache.org/r/7659/
>>
>>
>> On 11/16/2012 01:16 PM, Mohit Anchlia wrote:
>>
>> Another question I had was about rollover. What's the best way to
>> rollover files in reasonable timeframe? For instance our path is
>> YY/MM/DD/HH so every hour there is new file and the -1 hr is just sitting
>> with .tmp and it takes sometimes even hour before .tmp is closed and
>> renamed to .snappy. In this situation is there a way to tell flume to
>> rollover files sooner based on some idle time limit?
>>
>> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>>
>>> Thanks Mike it makes sense. Anyway I can help?
>>>
>>>
>>> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi Mohit, this is a complicated issue. I've filed
>>>> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
>>>>
>>>> In short, it would require a non-trivial amount of work to implement
>>>> this, and it would need to be done carefully. I agree that it would be
>>>> better if Flume handled this case more gracefully than it does today.
>>>> Today, Flume assumes that you have some job that would go and clean up the
>>>> .tmp files as needed, and that you understand that they could be partially
>>>> written if a crash occurred.
>>>>
>>>> Regards,
>>>> Mike
>>>>
>>>>
>>>> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> What we are seeing is that if flume gets killed either because of
>>>>> server failure or other reasons, it keeps around the .tmp file. Sometimes
>>>>> for whatever reasons .tmp file is not readable. Is there a way to rollover
>>>>> .tmp file more gracefully?
>>>>>
>>>>
>>>>
>>>
>>
>>
>
+
Mohit Anchlia 2012-11-20, 20:53
-
Re: .tmp in hdfs sinkBrock Noland 2012-11-20, 20:56
We are currently voting on a 1.3.0 RC on the dev@ list:
http://s.apache.org/OQ0W
You don't have to be a committer to vote! :)
Brock
On Tue, Nov 20, 2012 at 2:53 PM, Mohit Anchlia <[EMAIL PROTECTED]> wrote:
> Thanks a lot!! Now with this what should be the expected behaviour? After
> file is closed a new file is created for writes that come after closing the
> file?
>
> Thanks again for committing this change. Do you know when 1.3.0 is out? I am
> currently using the snapshot version of 1.3.0
>
> On Tue, Nov 20, 2012 at 11:16 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>>
>> Mohit,
>> FLUME-1660 is now committed and it will be in 1.3.0. In the case where you
>> are using 1.2.0, I suggest running with hdfs.rollInterval set so the files
>> will roll normally.
>>
>> Regards,
>> Mike
>>
>>
>> On Thu, Nov 15, 2012 at 11:23 PM, Juhani Connolly
>> <[EMAIL PROTECTED]> wrote:
>>>
>>> I am actually working on a patch for exactly this, refer to FLUME-1660
>>>
>>> The patch is on review board right now, I fixed a corner case issue that
>>> came up with unit testing, but the implementation is not really to my
>>> satisfaction. If you are interested please have a look and add your opinion.
>>>
>>> https://issues.apache.org/jira/browse/FLUME-1660
>>> https://reviews.apache.org/r/7659/
>>>
>>>
>>> On 11/16/2012 01:16 PM, Mohit Anchlia wrote:
>>>
>>> Another question I had was about rollover. What's the best way to
>>> rollover files in reasonable timeframe? For instance our path is YY/MM/DD/HH
>>> so every hour there is new file and the -1 hr is just sitting with .tmp and
>>> it takes sometimes even hour before .tmp is closed and renamed to .snappy.
>>> In this situation is there a way to tell flume to rollover files sooner
>>> based on some idle time limit?
>>>
>>> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <[EMAIL PROTECTED]>
>>> wrote:
>>>>
>>>> Thanks Mike it makes sense. Anyway I can help?
>>>>
>>>>
>>>> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>> Hi Mohit, this is a complicated issue. I've filed
>>>>> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
>>>>>
>>>>> In short, it would require a non-trivial amount of work to implement
>>>>> this, and it would need to be done carefully. I agree that it would be
>>>>> better if Flume handled this case more gracefully than it does today. Today,
>>>>> Flume assumes that you have some job that would go and clean up the .tmp
>>>>> files as needed, and that you understand that they could be partially
>>>>> written if a crash occurred.
>>>>>
>>>>> Regards,
>>>>> Mike
>>>>>
>>>>>
>>>>> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>>
>>>>>> What we are seeing is that if flume gets killed either because of
>>>>>> server failure or other reasons, it keeps around the .tmp file. Sometimes
>>>>>> for whatever reasons .tmp file is not readable. Is there a way to rollover
>>>>>> .tmp file more gracefully?
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
http://s.apache.org/OQ0W
You don't have to be a committer to vote! :)
Brock
On Tue, Nov 20, 2012 at 2:53 PM, Mohit Anchlia <[EMAIL PROTECTED]> wrote:
> Thanks a lot!! Now with this what should be the expected behaviour? After
> file is closed a new file is created for writes that come after closing the
> file?
>
> Thanks again for committing this change. Do you know when 1.3.0 is out? I am
> currently using the snapshot version of 1.3.0
>
> On Tue, Nov 20, 2012 at 11:16 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>>
>> Mohit,
>> FLUME-1660 is now committed and it will be in 1.3.0. In the case where you
>> are using 1.2.0, I suggest running with hdfs.rollInterval set so the files
>> will roll normally.
>>
>> Regards,
>> Mike
>>
>>
>> On Thu, Nov 15, 2012 at 11:23 PM, Juhani Connolly
>> <[EMAIL PROTECTED]> wrote:
>>>
>>> I am actually working on a patch for exactly this, refer to FLUME-1660
>>>
>>> The patch is on review board right now, I fixed a corner case issue that
>>> came up with unit testing, but the implementation is not really to my
>>> satisfaction. If you are interested please have a look and add your opinion.
>>>
>>> https://issues.apache.org/jira/browse/FLUME-1660
>>> https://reviews.apache.org/r/7659/
>>>
>>>
>>> On 11/16/2012 01:16 PM, Mohit Anchlia wrote:
>>>
>>> Another question I had was about rollover. What's the best way to
>>> rollover files in reasonable timeframe? For instance our path is YY/MM/DD/HH
>>> so every hour there is new file and the -1 hr is just sitting with .tmp and
>>> it takes sometimes even hour before .tmp is closed and renamed to .snappy.
>>> In this situation is there a way to tell flume to rollover files sooner
>>> based on some idle time limit?
>>>
>>> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <[EMAIL PROTECTED]>
>>> wrote:
>>>>
>>>> Thanks Mike it makes sense. Anyway I can help?
>>>>
>>>>
>>>> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>> Hi Mohit, this is a complicated issue. I've filed
>>>>> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
>>>>>
>>>>> In short, it would require a non-trivial amount of work to implement
>>>>> this, and it would need to be done carefully. I agree that it would be
>>>>> better if Flume handled this case more gracefully than it does today. Today,
>>>>> Flume assumes that you have some job that would go and clean up the .tmp
>>>>> files as needed, and that you understand that they could be partially
>>>>> written if a crash occurred.
>>>>>
>>>>> Regards,
>>>>> Mike
>>>>>
>>>>>
>>>>> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>>
>>>>>> What we are seeing is that if flume gets killed either because of
>>>>>> server failure or other reasons, it keeps around the .tmp file. Sometimes
>>>>>> for whatever reasons .tmp file is not readable. Is there a way to rollover
>>>>>> .tmp file more gracefully?
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
+
Brock Noland 2012-11-20, 20:56
-
Re: .tmp in hdfs sinkMike Percy 2012-11-20, 23:11
Mohit,
No problem, but Juhani did all the work. :)
The behavior is that you can configure an HDFS sink to close a file if it
hasn't gotten any writes in some time. After it's been idle for 5 minutes
or something, it gets closed. If you get a "late" event that goes to the
same path after the file is closed, it will just create a new file in the
same path as usual.
Regards,
Mike
On Tue, Nov 20, 2012 at 12:56 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
> We are currently voting on a 1.3.0 RC on the dev@ list:
>
> http://s.apache.org/OQ0W
>
> You don't have to be a committer to vote! :)
>
> Brock
>
> On Tue, Nov 20, 2012 at 2:53 PM, Mohit Anchlia <[EMAIL PROTECTED]>
> wrote:
> > Thanks a lot!! Now with this what should be the expected behaviour? After
> > file is closed a new file is created for writes that come after closing
> the
> > file?
> >
> > Thanks again for committing this change. Do you know when 1.3.0 is out?
> I am
> > currently using the snapshot version of 1.3.0
> >
> > On Tue, Nov 20, 2012 at 11:16 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
> >>
> >> Mohit,
> >> FLUME-1660 is now committed and it will be in 1.3.0. In the case where
> you
> >> are using 1.2.0, I suggest running with hdfs.rollInterval set so the
> files
> >> will roll normally.
> >>
> >> Regards,
> >> Mike
> >>
> >>
> >> On Thu, Nov 15, 2012 at 11:23 PM, Juhani Connolly
> >> <[EMAIL PROTECTED]> wrote:
> >>>
> >>> I am actually working on a patch for exactly this, refer to FLUME-1660
> >>>
> >>> The patch is on review board right now, I fixed a corner case issue
> that
> >>> came up with unit testing, but the implementation is not really to my
> >>> satisfaction. If you are interested please have a look and add your
> opinion.
> >>>
> >>> https://issues.apache.org/jira/browse/FLUME-1660
> >>> https://reviews.apache.org/r/7659/
> >>>
> >>>
> >>> On 11/16/2012 01:16 PM, Mohit Anchlia wrote:
> >>>
> >>> Another question I had was about rollover. What's the best way to
> >>> rollover files in reasonable timeframe? For instance our path is
> YY/MM/DD/HH
> >>> so every hour there is new file and the -1 hr is just sitting with
> .tmp and
> >>> it takes sometimes even hour before .tmp is closed and renamed to
> .snappy.
> >>> In this situation is there a way to tell flume to rollover files sooner
> >>> based on some idle time limit?
> >>>
> >>> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <[EMAIL PROTECTED]
> >
> >>> wrote:
> >>>>
> >>>> Thanks Mike it makes sense. Anyway I can help?
> >>>>
> >>>>
> >>>> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]>
> wrote:
> >>>>>
> >>>>> Hi Mohit, this is a complicated issue. I've filed
> >>>>> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
> >>>>>
> >>>>> In short, it would require a non-trivial amount of work to implement
> >>>>> this, and it would need to be done carefully. I agree that it would
> be
> >>>>> better if Flume handled this case more gracefully than it does
> today. Today,
> >>>>> Flume assumes that you have some job that would go and clean up the
> .tmp
> >>>>> files as needed, and that you understand that they could be partially
> >>>>> written if a crash occurred.
> >>>>>
> >>>>> Regards,
> >>>>> Mike
> >>>>>
> >>>>>
> >>>>> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <
> [EMAIL PROTECTED]>
> >>>>> wrote:
> >>>>>>
> >>>>>> What we are seeing is that if flume gets killed either because of
> >>>>>> server failure or other reasons, it keeps around the .tmp file.
> Sometimes
> >>>>>> for whatever reasons .tmp file is not readable. Is there a way to
> rollover
> >>>>>> .tmp file more gracefully?
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
No problem, but Juhani did all the work. :)
The behavior is that you can configure an HDFS sink to close a file if it
hasn't gotten any writes in some time. After it's been idle for 5 minutes
or something, it gets closed. If you get a "late" event that goes to the
same path after the file is closed, it will just create a new file in the
same path as usual.
Regards,
Mike
On Tue, Nov 20, 2012 at 12:56 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
> We are currently voting on a 1.3.0 RC on the dev@ list:
>
> http://s.apache.org/OQ0W
>
> You don't have to be a committer to vote! :)
>
> Brock
>
> On Tue, Nov 20, 2012 at 2:53 PM, Mohit Anchlia <[EMAIL PROTECTED]>
> wrote:
> > Thanks a lot!! Now with this what should be the expected behaviour? After
> > file is closed a new file is created for writes that come after closing
> the
> > file?
> >
> > Thanks again for committing this change. Do you know when 1.3.0 is out?
> I am
> > currently using the snapshot version of 1.3.0
> >
> > On Tue, Nov 20, 2012 at 11:16 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
> >>
> >> Mohit,
> >> FLUME-1660 is now committed and it will be in 1.3.0. In the case where
> you
> >> are using 1.2.0, I suggest running with hdfs.rollInterval set so the
> files
> >> will roll normally.
> >>
> >> Regards,
> >> Mike
> >>
> >>
> >> On Thu, Nov 15, 2012 at 11:23 PM, Juhani Connolly
> >> <[EMAIL PROTECTED]> wrote:
> >>>
> >>> I am actually working on a patch for exactly this, refer to FLUME-1660
> >>>
> >>> The patch is on review board right now, I fixed a corner case issue
> that
> >>> came up with unit testing, but the implementation is not really to my
> >>> satisfaction. If you are interested please have a look and add your
> opinion.
> >>>
> >>> https://issues.apache.org/jira/browse/FLUME-1660
> >>> https://reviews.apache.org/r/7659/
> >>>
> >>>
> >>> On 11/16/2012 01:16 PM, Mohit Anchlia wrote:
> >>>
> >>> Another question I had was about rollover. What's the best way to
> >>> rollover files in reasonable timeframe? For instance our path is
> YY/MM/DD/HH
> >>> so every hour there is new file and the -1 hr is just sitting with
> .tmp and
> >>> it takes sometimes even hour before .tmp is closed and renamed to
> .snappy.
> >>> In this situation is there a way to tell flume to rollover files sooner
> >>> based on some idle time limit?
> >>>
> >>> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <[EMAIL PROTECTED]
> >
> >>> wrote:
> >>>>
> >>>> Thanks Mike it makes sense. Anyway I can help?
> >>>>
> >>>>
> >>>> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]>
> wrote:
> >>>>>
> >>>>> Hi Mohit, this is a complicated issue. I've filed
> >>>>> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
> >>>>>
> >>>>> In short, it would require a non-trivial amount of work to implement
> >>>>> this, and it would need to be done carefully. I agree that it would
> be
> >>>>> better if Flume handled this case more gracefully than it does
> today. Today,
> >>>>> Flume assumes that you have some job that would go and clean up the
> .tmp
> >>>>> files as needed, and that you understand that they could be partially
> >>>>> written if a crash occurred.
> >>>>>
> >>>>> Regards,
> >>>>> Mike
> >>>>>
> >>>>>
> >>>>> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <
> [EMAIL PROTECTED]>
> >>>>> wrote:
> >>>>>>
> >>>>>> What we are seeing is that if flume gets killed either because of
> >>>>>> server failure or other reasons, it keeps around the .tmp file.
> Sometimes
> >>>>>> for whatever reasons .tmp file is not readable. Is there a way to
> rollover
> >>>>>> .tmp file more gracefully?
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
+
Mike Percy 2012-11-20, 23:11
-
Re: .tmp in hdfs sinkMohit Anchlia 2012-11-20, 23:24
that's awesome!
On Tue, Nov 20, 2012 at 3:11 PM, Mike Percy <[EMAIL PROTECTED]> wrote:
> Mohit,
> No problem, but Juhani did all the work. :)
>
> The behavior is that you can configure an HDFS sink to close a file if it
> hasn't gotten any writes in some time. After it's been idle for 5 minutes
> or something, it gets closed. If you get a "late" event that goes to the
> same path after the file is closed, it will just create a new file in the
> same path as usual.
>
> Regards,
> Mike
>
>
> On Tue, Nov 20, 2012 at 12:56 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>
>> We are currently voting on a 1.3.0 RC on the dev@ list:
>>
>> http://s.apache.org/OQ0W
>>
>> You don't have to be a committer to vote! :)
>>
>> Brock
>>
>> On Tue, Nov 20, 2012 at 2:53 PM, Mohit Anchlia <[EMAIL PROTECTED]>
>> wrote:
>> > Thanks a lot!! Now with this what should be the expected behaviour?
>> After
>> > file is closed a new file is created for writes that come after closing
>> the
>> > file?
>> >
>> > Thanks again for committing this change. Do you know when 1.3.0 is out?
>> I am
>> > currently using the snapshot version of 1.3.0
>> >
>> > On Tue, Nov 20, 2012 at 11:16 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Mohit,
>> >> FLUME-1660 is now committed and it will be in 1.3.0. In the case where
>> you
>> >> are using 1.2.0, I suggest running with hdfs.rollInterval set so the
>> files
>> >> will roll normally.
>> >>
>> >> Regards,
>> >> Mike
>> >>
>> >>
>> >> On Thu, Nov 15, 2012 at 11:23 PM, Juhani Connolly
>> >> <[EMAIL PROTECTED]> wrote:
>> >>>
>> >>> I am actually working on a patch for exactly this, refer to FLUME-1660
>> >>>
>> >>> The patch is on review board right now, I fixed a corner case issue
>> that
>> >>> came up with unit testing, but the implementation is not really to my
>> >>> satisfaction. If you are interested please have a look and add your
>> opinion.
>> >>>
>> >>> https://issues.apache.org/jira/browse/FLUME-1660
>> >>> https://reviews.apache.org/r/7659/
>> >>>
>> >>>
>> >>> On 11/16/2012 01:16 PM, Mohit Anchlia wrote:
>> >>>
>> >>> Another question I had was about rollover. What's the best way to
>> >>> rollover files in reasonable timeframe? For instance our path is
>> YY/MM/DD/HH
>> >>> so every hour there is new file and the -1 hr is just sitting with
>> .tmp and
>> >>> it takes sometimes even hour before .tmp is closed and renamed to
>> .snappy.
>> >>> In this situation is there a way to tell flume to rollover files
>> sooner
>> >>> based on some idle time limit?
>> >>>
>> >>> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <
>> [EMAIL PROTECTED]>
>> >>> wrote:
>> >>>>
>> >>>> Thanks Mike it makes sense. Anyway I can help?
>> >>>>
>> >>>>
>> >>>> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]>
>> wrote:
>> >>>>>
>> >>>>> Hi Mohit, this is a complicated issue. I've filed
>> >>>>> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
>> >>>>>
>> >>>>> In short, it would require a non-trivial amount of work to implement
>> >>>>> this, and it would need to be done carefully. I agree that it would
>> be
>> >>>>> better if Flume handled this case more gracefully than it does
>> today. Today,
>> >>>>> Flume assumes that you have some job that would go and clean up the
>> .tmp
>> >>>>> files as needed, and that you understand that they could be
>> partially
>> >>>>> written if a crash occurred.
>> >>>>>
>> >>>>> Regards,
>> >>>>> Mike
>> >>>>>
>> >>>>>
>> >>>>> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <
>> [EMAIL PROTECTED]>
>> >>>>> wrote:
>> >>>>>>
>> >>>>>> What we are seeing is that if flume gets killed either because of
>> >>>>>> server failure or other reasons, it keeps around the .tmp file.
>> Sometimes
>> >>>>>> for whatever reasons .tmp file is not readable. Is there a way to
>> rollover
>> >>>>>> .tmp file more gracefully?
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
On Tue, Nov 20, 2012 at 3:11 PM, Mike Percy <[EMAIL PROTECTED]> wrote:
> Mohit,
> No problem, but Juhani did all the work. :)
>
> The behavior is that you can configure an HDFS sink to close a file if it
> hasn't gotten any writes in some time. After it's been idle for 5 minutes
> or something, it gets closed. If you get a "late" event that goes to the
> same path after the file is closed, it will just create a new file in the
> same path as usual.
>
> Regards,
> Mike
>
>
> On Tue, Nov 20, 2012 at 12:56 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>
>> We are currently voting on a 1.3.0 RC on the dev@ list:
>>
>> http://s.apache.org/OQ0W
>>
>> You don't have to be a committer to vote! :)
>>
>> Brock
>>
>> On Tue, Nov 20, 2012 at 2:53 PM, Mohit Anchlia <[EMAIL PROTECTED]>
>> wrote:
>> > Thanks a lot!! Now with this what should be the expected behaviour?
>> After
>> > file is closed a new file is created for writes that come after closing
>> the
>> > file?
>> >
>> > Thanks again for committing this change. Do you know when 1.3.0 is out?
>> I am
>> > currently using the snapshot version of 1.3.0
>> >
>> > On Tue, Nov 20, 2012 at 11:16 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Mohit,
>> >> FLUME-1660 is now committed and it will be in 1.3.0. In the case where
>> you
>> >> are using 1.2.0, I suggest running with hdfs.rollInterval set so the
>> files
>> >> will roll normally.
>> >>
>> >> Regards,
>> >> Mike
>> >>
>> >>
>> >> On Thu, Nov 15, 2012 at 11:23 PM, Juhani Connolly
>> >> <[EMAIL PROTECTED]> wrote:
>> >>>
>> >>> I am actually working on a patch for exactly this, refer to FLUME-1660
>> >>>
>> >>> The patch is on review board right now, I fixed a corner case issue
>> that
>> >>> came up with unit testing, but the implementation is not really to my
>> >>> satisfaction. If you are interested please have a look and add your
>> opinion.
>> >>>
>> >>> https://issues.apache.org/jira/browse/FLUME-1660
>> >>> https://reviews.apache.org/r/7659/
>> >>>
>> >>>
>> >>> On 11/16/2012 01:16 PM, Mohit Anchlia wrote:
>> >>>
>> >>> Another question I had was about rollover. What's the best way to
>> >>> rollover files in reasonable timeframe? For instance our path is
>> YY/MM/DD/HH
>> >>> so every hour there is new file and the -1 hr is just sitting with
>> .tmp and
>> >>> it takes sometimes even hour before .tmp is closed and renamed to
>> .snappy.
>> >>> In this situation is there a way to tell flume to rollover files
>> sooner
>> >>> based on some idle time limit?
>> >>>
>> >>> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <
>> [EMAIL PROTECTED]>
>> >>> wrote:
>> >>>>
>> >>>> Thanks Mike it makes sense. Anyway I can help?
>> >>>>
>> >>>>
>> >>>> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]>
>> wrote:
>> >>>>>
>> >>>>> Hi Mohit, this is a complicated issue. I've filed
>> >>>>> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
>> >>>>>
>> >>>>> In short, it would require a non-trivial amount of work to implement
>> >>>>> this, and it would need to be done carefully. I agree that it would
>> be
>> >>>>> better if Flume handled this case more gracefully than it does
>> today. Today,
>> >>>>> Flume assumes that you have some job that would go and clean up the
>> .tmp
>> >>>>> files as needed, and that you understand that they could be
>> partially
>> >>>>> written if a crash occurred.
>> >>>>>
>> >>>>> Regards,
>> >>>>> Mike
>> >>>>>
>> >>>>>
>> >>>>> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <
>> [EMAIL PROTECTED]>
>> >>>>> wrote:
>> >>>>>>
>> >>>>>> What we are seeing is that if flume gets killed either because of
>> >>>>>> server failure or other reasons, it keeps around the .tmp file.
>> Sometimes
>> >>>>>> for whatever reasons .tmp file is not readable. Is there a way to
>> rollover
>> >>>>>> .tmp file more gracefully?
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
+
Mohit Anchlia 2012-11-20, 23:24
-
Re: .tmp in hdfs sinkAlexander Alten-Lorenz 2012-11-16, 06:51
I have made this in past per minute rolls (YY/MM/DD/HH/MM) and closed a sink after 30 secs. This matched in my cases mostly perfect. But depends on your use case.
Cheers,
Alex
On Nov 16, 2012, at 5:16 AM, Mohit Anchlia <[EMAIL PROTECTED]> wrote:
> Another question I had was about rollover. What's the best way to rollover
> files in reasonable timeframe? For instance our path is YY/MM/DD/HH so
> every hour there is new file and the -1 hr is just sitting with .tmp and it
> takes sometimes even hour before .tmp is closed and renamed to .snappy. In
> this situation is there a way to tell flume to rollover files sooner based
> on some idle time limit?
>
> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>
>> Thanks Mike it makes sense. Anyway I can help?
>>
>>
>> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>>
>>> Hi Mohit, this is a complicated issue. I've filed
>>> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
>>>
>>> In short, it would require a non-trivial amount of work to implement
>>> this, and it would need to be done carefully. I agree that it would be
>>> better if Flume handled this case more gracefully than it does today.
>>> Today, Flume assumes that you have some job that would go and clean up the
>>> .tmp files as needed, and that you understand that they could be partially
>>> written if a crash occurred.
>>>
>>> Regards,
>>> Mike
>>>
>>>
>>> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>>>
>>>> What we are seeing is that if flume gets killed either because of server
>>>> failure or other reasons, it keeps around the .tmp file. Sometimes for
>>>> whatever reasons .tmp file is not readable. Is there a way to rollover .tmp
>>>> file more gracefully?
>>>>
>>>
>>>
>>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Cheers,
Alex
On Nov 16, 2012, at 5:16 AM, Mohit Anchlia <[EMAIL PROTECTED]> wrote:
> Another question I had was about rollover. What's the best way to rollover
> files in reasonable timeframe? For instance our path is YY/MM/DD/HH so
> every hour there is new file and the -1 hr is just sitting with .tmp and it
> takes sometimes even hour before .tmp is closed and renamed to .snappy. In
> this situation is there a way to tell flume to rollover files sooner based
> on some idle time limit?
>
> On Thu, Nov 15, 2012 at 8:14 PM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>
>> Thanks Mike it makes sense. Anyway I can help?
>>
>>
>> On Thu, Nov 15, 2012 at 11:54 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>>
>>> Hi Mohit, this is a complicated issue. I've filed
>>> https://issues.apache.org/jira/browse/FLUME-1714 to track it.
>>>
>>> In short, it would require a non-trivial amount of work to implement
>>> this, and it would need to be done carefully. I agree that it would be
>>> better if Flume handled this case more gracefully than it does today.
>>> Today, Flume assumes that you have some job that would go and clean up the
>>> .tmp files as needed, and that you understand that they could be partially
>>> written if a crash occurred.
>>>
>>> Regards,
>>> Mike
>>>
>>>
>>> On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>>>
>>>> What we are seeing is that if flume gets killed either because of server
>>>> failure or other reasons, it keeps around the .tmp file. Sometimes for
>>>> whatever reasons .tmp file is not readable. Is there a way to rollover .tmp
>>>> file more gracefully?
>>>>
>>>
>>>
>>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
+
Alexander Alten-Lorenz 2012-11-16, 06:51
