Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume-NG -- 2 Sources and a single Sink


Copy link to this message
-
Re: Flume-NG -- 2 Sources and a single Sink
Prashanth,

I am glad things are working out for you.

Good luck with your project.

On Wed, Apr 3, 2013 at 12:06 AM, Babu, Prashanth <[EMAIL PROTECTED]
> wrote:

>  Thanks for your inputs Israel.****
>
> As of now, both my sources are using separate channels [Memory] and my
> custom sink is just getting all the events and pushing them to a flat file
> on the disk.****
>
> I am trying to learn and understand Flume-NG with this prototype.****
>
> I will go thru the Flume User Guide as well.****
>
> ** **
>
> Regards,****
>
> Prashanth.****
>
> ** **
>
> *From:* Israel Ekpo [mailto:[EMAIL PROTECTED]]
> *Sent:* Tuesday, April 02, 2013 4:34 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: Flume-NG -- 2 Sources and a single Sink****
>
> ** **
>
> Checkout the user guide for Flume NG instead
>
> http://flume.apache.org/FlumeUserGuide.html
>
> I would suggest that you use separate channels (preferably the
> FileChannel) for each source.
>
> I am not sure what your custom sink looks like, but if it capable of
> handling multiple incoming requests simultaneously like MongoDB, Elastic
> Search, Apache Solr, MySQL or CouchBase you should still be able to set
> them up as separate sinks but technically all the events are going to the
> same centralized store.
>
> The user guide is more up to date and contains more examples that could
> help you decide which way to go.****
>
> ** **
>
> Review the various examples and data flow designs.****
>
> On Tue, Apr 2, 2013 at 4:25 AM, Babu, Prashanth <
> [EMAIL PROTECTED]> wrote:****
>
> Hi,
>
> I have been using Flume-OG till now and currently I am evaluating Flume-NG.
> I have followed the guidelines in
> http://flume.apache.org/FlumeDeveloperGuide.html.
> In my use case, I have 2 Sources on a machine, but a single custom sink.
> 2 Sources are tails on 2 different files which are configured in
> flume.conf.
> I have tried giving the same channel for both the sources, but my sink is
> getting data only from one of the sources.
> Is there any way wherein I can get data from both the sources with a
> single sink, as is required in my case?
> Please let me know.
> TIA.
>
> Regards,
> Prashanth.
>
> ______________________________________________________________________
> Disclaimer:This email and any attachments are sent in strictest confidence
> for the sole use of the addressee and may contain legally privileged,
> confidential, and proprietary data.  If you are not the intended recipient,
> please advise the sender by replying promptly to this email and then delete
> and destroy this email and any attachments without any further use, copying
> or forwarding****
>
> ** **
>
> ______________________________________________________________________
> Disclaimer:This email and any attachments are sent in strictest confidence
> for the sole use of the addressee and may contain legally privileged,
> confidential, and proprietary data. If you are not the intended recipient,
> please advise the sender by replying promptly to this email and then delete
> and destroy this email and any attachments without any further use, copying
> or forwarding
>