Hello,

could sameone please tell me why HDFSEventSink returns Status.BACKOFF in
case of IOException? Because of this implementation, FailoverSinkProcessor
will never push event to next sink in case of any hdfs failure.

Another thing is that BucketWriters should be removed from sfWriters map
after IOException, can anyone please confirm that?

--
Regards,
J. Grabowski

Under what circumstances are you seeing an IOException? Can you post the
full stack trace?

Best,
Mike
On Tue, Dec 11, 2012 at 5:22 AM, Jaroslaw Grabowski
<[EMAIL PROTECTED]>wrote:

> Hello,
>
> could sameone please tell me why HDFSEventSink returns Status.BACKOFF in
> case of IOException? Because of this implementation, FailoverSinkProcessor
> will never push event to next sink in case of any hdfs failure.
>
> Another thing is that BucketWriters should be removed from sfWriters map
> after IOException, can anyone please confirm that?
>
> --
> Regards,
> J. Grabowski
>

It is easy to reproduce. For example IOException will be thrown from
BucketWriter after all DataNodes in cluster become unavailable. I believe
that in that case, BucketWriter won't return to operational state even if
cluster is back and ready to use. It is probably a good idea to discard
BucketWriters after IOException. Do you agree?

I've filed a jira with stack trace in attached file:
https://issues.apache.org/jira/browse/FLUME-1779

Regards,
J. Grabowski
2012/12/12 Mike Percy <[EMAIL PROTECTED]>

> Under what circumstances are you seeing an IOException? Can you post the
> full stack trace?
>
> Best,
> Mike
>
>
> On Tue, Dec 11, 2012 at 5:22 AM, Jaroslaw Grabowski
> <[EMAIL PROTECTED]>wrote:
>
> > Hello,
> >
> > could sameone please tell me why HDFSEventSink returns Status.BACKOFF in
> > case of IOException? Because of this implementation,
> FailoverSinkProcessor
> > will never push event to next sink in case of any hdfs failure.
> >
> > Another thing is that BucketWriters should be removed from sfWriters map
> > after IOException, can anyone please confirm that?
> >
> > --
> > Regards,
> > J. Grabowski
> >
>

--
Pozdrawiam,
Jarosław Grabowski

Hi,

I am not an expert in the FailoverSinkProcessor but this sounds like a
bug to me. Can you file a jira?

Brock

On Tue, Dec 11, 2012 at 7:22 AM, Jaroslaw Grabowski
<[EMAIL PROTECTED]> wrote:
> Hello,
>
> could sameone please tell me why HDFSEventSink returns Status.BACKOFF in
> case of IOException? Because of this implementation, FailoverSinkProcessor
> will never push event to next sink in case of any hdfs failure.
>
> Another thing is that BucketWriters should be removed from sfWriters map
> after IOException, can anyone please confirm that?
>
> --
> Regards,
> J. Grabowski

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/