Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Source and Sink on different machine


Copy link to this message
-
Re: Source and Sink on different machine
Juhani Connolly 2012-12-28, 08:09
You connect up the various machines using avro sink/source.

You set up 2 & 3 with avro source, a channel and hdfs sink.

Then you set up machine one with whatever source, channel and two avro
sinks, and a sink
processor(http://flume.apache.org/FlumeUserGuide.html#flume-sink-processors)
of your personal preference. If you use failover when the main sink
fails, events will go through the backup instead.

On 12/28/2012 04:53 PM, Abhijeet Pathak wrote:
>
> I want to setup a failover configuration.
>
> On Machine1 I've a source.
>
> On Machine2 & Machine3, I have (hbase) sinks.
>
> I want to configure them in failover mode. That is, by default events
> should go to sink on Machine2's sink. But if Machine2 fails then they
> should automatically go to Machine3's sink.
>
> Is that possible in Flume-NG? How?
>
> Thanks,
>
> Abhijeet
>
> *From:*Nitin Pawar [mailto:[EMAIL PROTECTED]]
> *Sent:* Friday, December 28, 2012 1:18 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: Source and Sink on different machine
>
> if I understood it correctly, you want to put your log data into hbase
> which is hosted remotely?
>
> yes thats easily doable.
>
> Flume-NG does everything alone instead of older master-slave
> architecture. It has currently source channel and sink. you can
> collect logs on a machine via an agent and using same agent you can
> sink them into hbase  on a remote host
>
> On Fri, Dec 28, 2012 at 12:06 PM, Abhijeet Pathak
> <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
> Hi,
>
> It was possible to have source and sinks on different machines using
> Flume 0.9x.
>
> Is it possible to have source and sinks on different machines in
> Flume-NG (1.3.0)?
>
> I want to use tail source on one machine and hbase sink on another
> machine.
>
> How should I go about it?
>
> Thanks,
>
> Abhijeet
>
>
> This message contains information that may be privileged or
> confidential and is the property of the KPIT Cummins Infosystems Ltd.
> It is intended only for the person to whom it is addressed. If you are
> not the intended recipient, you are not authorized to read, print,
> retain copy, disseminate, distribute, or use this message or any part
> thereof. If you receive this message in error, please notify the
> sender immediately and delete all copies of this message. KPIT Cummins
> Infosystems Ltd. does not accept any liability for virus infected mails.
>
>
>
> --
> Nitin Pawar
>
>
> This message contains information that may be privileged or
> confidential and is the property of the KPIT Cummins Infosystems Ltd.
> It is intended only for the person to whom it is addressed. If you are
> not the intended recipient, you are not authorized to read, print,
> retain copy, disseminate, distribute, or use this message or any part
> thereof. If you receive this message in error, please notify the
> sender immediately and delete all copies of this message. KPIT Cummins
> Infosystems Ltd. does not accept any liability for virus infected mails.