Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume hops behind HAProxy


Copy link to this message
-
Re: Flume hops behind HAProxy
That will probably work. From my earlier mail:

"I assume it would be connection based round robin?  Might work just
fine, but probably best to the use built-in support."

On Thu, Nov 15, 2012 at 11:27 AM, Rahul Ravindran <[EMAIL PROTECTED]> wrote:
> HAProxy has a TCP mode where it round robins TCP connections. Does it need
> to understand something specific about the wire protocol used by Flume?
>
> ________________________________
> From: Brock Noland <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]; Rahul Ravindran <[EMAIL PROTECTED]>
> Sent: Wednesday, November 14, 2012 6:20 PM
>
> Subject: Re: Flume hops behind HAProxy
>
>> It would be round robin but not sticky sessions( so each request could
>> Goto any random flume hop)
>
> Does HAProxy understand the protocol?  To round robin requests like
> this it needs to understand the communication protocol, which I
> suppose would work if you were using the HTTPSource.
>
> On Wed, Nov 14, 2012 at 4:46 PM, Rahul Ravindran <[EMAIL PROTECTED]> wrote:
>> HAProxy is widely deployed already in our deployment and Ops is familiar
>> with dealing with it for hosts which go down etc.
>>
>> ________________________________
>> From: "Camp, Roy" <[EMAIL PROTECTED]>
>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>> Sent: Wednesday, November 14, 2012 2:15 PM
>> Subject: RE: Flume hops behind HAProxy
>>
>> Out of curiosity, what is the use case vs using the built in load
>> balancing?
>>
>>
>>
>> -----Original Message-----
>> From: Rahul Ravindran [mailto:[EMAIL PROTECTED]]
>> Sent: Wednesday, November 14, 2012 1:49 PM
>> To: [EMAIL PROTECTED]
>> Cc: [EMAIL PROTECTED]
>> Subject: Re: Flume hops behind HAProxy
>>
>> It would be round robin but not sticky sessions( so each request could
>> Goto
>> any random flume hop)
>>
>> Sent from my phone.Excuse the terseness.
>>
>> On Nov 14, 2012, at 1:33 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>>
>>> I assume it would be connection based round robin?  Might work just
>>> fine, but probably best to the use built-in support.
>>>
>>> On Wed, Nov 14, 2012 at 2:46 PM, Rahul Ravindran <[EMAIL PROTECTED]>
>>> wrote:
>>>> Resending given I sent it during off-hours.
>>>>
>>>> ________________________________
>>>> From: Rahul Ravindran <[EMAIL PROTECTED]>
>>>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>>> Sent: Tuesday, November 13, 2012 5:52 PM
>>>> Subject: Flume hops behind HAProxy
>>>>
>>>> Hi,
>>>>  Before I try it, I wanted to check if there were any known issues
>>>> with this. We will have multiple flume agents sending an Avro stream
>>>> each to a smaller set of intermediate flume hops. Are there any
>>>> issues/concerns around having the flume agents send their streams to
>>>> an HAProxy which will round robin between the different flume hops.
>>>> Any issue around the transaction mechanism with this setup?
>>>>
>>>> I know that there is a selector mechanism in Flume to do this, but
>>>> our operations extensively use HAProxy, and are most familiar with it.
>>>>
>>>> Thanks,
>>>> ~Rahul.
>>>
>>>
>>>
>>> --
>>> Apache MRUnit - Unit testing MapReduce -
>>> http://incubator.apache.org/mrunit/
>>
>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
>
>

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/