Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Configuring FLUME to use FailOverSinkProcessor...


Copy link to this message
-
Re: Configuring FLUME to use FailOverSinkProcessor...
So if this thing works what are people's thoughts about using it for PROD
envs...

Jarek, the reason i had hdfs data sink locations same for both HDFS sinks
was to ensure that the spooled data also makes to the final location...so
the test to try out would be to check all numbers generated by the seqeunce
generator source are available once the tests are finished...

btw, the avro-sink & avro source combination for file channel is a little
heavy as it's local..perhaps something lighter could be better.

* I turned off name node, event's were not stored in file channel as
expected
>> here you meant they were indeed saved into file channel and drained from
memory channel right??

Thanks,
- Inder

On Fri, Jul 13, 2012 at 7:18 PM, Inder Pall <[EMAIL PROTECTED]> wrote:

> Jarek,
>
> thanks for taking out time to try this..yeah i meant mem channel to be
> used first and then file channel for some reason i thought lower number
> means higher priority pardon the ignorance of not looking at the
> documentation.....
>
> i have a dated code-base of trunk around 10-12 days...the agent comes up
> fine but the sequence generator source wasn't sending any events...or
> actually i didn't see anything in the logs....
>
>
> Thanks,
>  - Inder
>
>
> On Fri, Jul 13, 2012 at 3:30 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>
>> Hi Inder,
>> could you please advise what exactly was (or wasn't) happening to you?
>> I've tried your configuration file on current trunk and it was working for
>> me out of the box.
>>
>> I was playing with it to see if your idea will work and I've ended up
>> with attached configuration. It contains just few modification to yours:
>>
>> * I've swapped priorities to firstly use the memory channel and fail over
>> to file channel
>> * I've renamed target file prefixes to distinguish source of the events
>> (memory or file channel)
>>
>> Scenario that I've run:
>>
>> * I turned on the flume agents, event's were correctly saved on HDFS
>> * I turned off name node, event's were not stored in file channel as
>> expected
>>
>> I'm not sure why the observed behaviour was different than expected and
>> 'I'll investigate that later. Meantime, could you describe what exactly was
>> happening to you?
>>
>> Jarcec
>>
>> On Thu, Jul 12, 2012 at 04:11:08PM +0200, Jarek Jarcec Cecho wrote:
>> > I'm just looking on that sir.
>> >
>> > Jarcec
>> >
>> > On Thu, Jul 12, 2012 at 07:34:31PM +0530, Inder Pall wrote:
>> > > Folks,
>> > >
>> > > for some reason updating the JIRA isn't trigerring an email....
>> > > I need feedback from FLUME DEVS on
>> > > FLUME-1045<
>> https://issues.apache.org/jira/browse/FLUME-1045?focusedCommentId=13412737&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13412737
>> >
>> > > wherein
>> > > i am trying to use failover sinkprocessor and a combination of file
>> and
>> > > memory channel to achieve scribe like spooling/de-spooling.....facing
>> some
>> > > issues here....
>> > >
>> > > --
>> > > Thanks,
>> > > - Inder
>> > >   Tech Platforms @Inmobi
>> > >   Linkedin - http://goo.gl/eR4Ub
>>
>>
>>
>
>
> --
> Thanks,
> - Inder
>   Tech Platforms @Inmobi
>   Linkedin - http://goo.gl/eR4Ub
>

--
Thanks,
- Inder
  Tech Platforms @Inmobi
  Linkedin - http://goo.gl/eR4Ub