Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - HTTP Source


Copy link to this message
-
Re: HTTP Source
Hari Shreedharan 2012-11-06, 18:55
Hi Nathaniel,

When I wrote this (and while it was being reviewed), we concluded it was not a good idea to expose the response object to the custom handler which can be plugged in because the handler can flush the buffer of the response object (which also can be flushed just because it is full) and then we could hit some exception (like channel being full) causing inconsistencies in the response. This source is not meant to be used for HTTP based RPC etc, this is to be used to send the data to the source and then wait for success or failure - on success send next event, on failure - back off for a while and retry (or similar logic).

For RPC etc, it might be best to write a source specific to your use-case. In case of this source bundled with Flume, we had to make it general enough to be useful but cautious enough to not make it confuse applications sending the data into Flume.

Hope this helps.

Thanks,
Hari

--
Hari Shreedharan
On Monday, November 5, 2012 at 10:09 AM, Nathaniel Auvil wrote:

> One thing i do not see with this HTTPSource is any way to customize the response.
>
>
> On Mon, Nov 5, 2012 at 10:58 AM, Brock Noland <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > Very soon. I am going to wait for the latest rev of this patch
> > (https://reviews.apache.org/r/6377/) to be reviewed in hopes it can be
> > committed and either way I will build the RC.
> >
> > Brock
> >
> > On Mon, Nov 5, 2012 at 9:47 AM, Nathaniel Auvil
> > <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > is there a target date for 1.3?
> > >
> > >
> > > On Mon, Nov 5, 2012 at 10:34 AM, Brock Noland <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > >>
> > >> There is actually an HTTP Source committed to the 1.3 branch:
> > >>
> > >>
> > >> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c
> > >>
> > >> Otherwise I would shade or jarjar your HTTP Source jars so they don't
> > >> conflict.
> > >>
> > >> Brock
> > >>
> > >> On Mon, Nov 5, 2012 at 9:31 AM, Nathaniel Auvil
> > >> <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > >> > i am working with Flume 1.2.0 and creating my own HTTP source by
> > >> > embedding
> > >> > Tomcat. I am having several issues.  Has anyone ever gotten this to
> > >> > work?
> > >> > It looks like my latest issue is a servlet-api conflict due to an
> > >> > ancient
> > >> > Jetty 6 version on the Flume classpath
> > >>
> > >>
> > >>
> > >> --
> > >> Apache MRUnit - Unit testing MapReduce -
> > >> http://incubator.apache.org/mrunit/
> > >
> > >
> >
> >
> >
> > --
> > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
>