i am working with Flume 1.2.0 and creating my own HTTP source by embedding
Tomcat. I am having several issues.  Has anyone ever gotten this to work?
 It looks like my latest issue is a servlet-api conflict due to an ancient
Jetty 6 version on the Flume classpath

There is actually an HTTP Source committed to the 1.3 branch:

https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c

Otherwise I would shade or jarjar your HTTP Source jars so they don't conflict.

Brock

On Mon, Nov 5, 2012 at 9:31 AM, Nathaniel Auvil
<[EMAIL PROTECTED]> wrote:
> i am working with Flume 1.2.0 and creating my own HTTP source by embedding
> Tomcat. I am having several issues.  Has anyone ever gotten this to work?
> It looks like my latest issue is a servlet-api conflict due to an ancient
> Jetty 6 version on the Flume classpath

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

is there a target date for 1.3?
On Mon, Nov 5, 2012 at 10:34 AM, Brock Noland <[EMAIL PROTECTED]> wrote:

> There is actually an HTTP Source committed to the 1.3 branch:
>
>
> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c
>
> Otherwise I would shade or jarjar your HTTP Source jars so they don't
> conflict.
>
> Brock
>
> On Mon, Nov 5, 2012 at 9:31 AM, Nathaniel Auvil
> <[EMAIL PROTECTED]> wrote:
> > i am working with Flume 1.2.0 and creating my own HTTP source by
> embedding
> > Tomcat. I am having several issues.  Has anyone ever gotten this to work?
> > It looks like my latest issue is a servlet-api conflict due to an ancient
> > Jetty 6 version on the Flume classpath
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>

Very soon. I am going to wait for the latest rev of this patch
(https://reviews.apache.org/r/6377/) to be reviewed in hopes it can be
committed and either way I will build the RC.

Brock

On Mon, Nov 5, 2012 at 9:47 AM, Nathaniel Auvil
<[EMAIL PROTECTED]> wrote:
> is there a target date for 1.3?
>
>
> On Mon, Nov 5, 2012 at 10:34 AM, Brock Noland <[EMAIL PROTECTED]> wrote:
>>
>> There is actually an HTTP Source committed to the 1.3 branch:
>>
>>
>> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c
>>
>> Otherwise I would shade or jarjar your HTTP Source jars so they don't
>> conflict.
>>
>> Brock
>>
>> On Mon, Nov 5, 2012 at 9:31 AM, Nathaniel Auvil
>> <[EMAIL PROTECTED]> wrote:
>> > i am working with Flume 1.2.0 and creating my own HTTP source by
>> > embedding
>> > Tomcat. I am having several issues.  Has anyone ever gotten this to
>> > work?
>> > It looks like my latest issue is a servlet-api conflict due to an
>> > ancient
>> > Jetty 6 version on the Flume classpath
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>
>

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

One thing i do not see with this HTTPSource is any way to customize the
response.
On Mon, Nov 5, 2012 at 10:58 AM, Brock Noland <[EMAIL PROTECTED]> wrote:

> Very soon. I am going to wait for the latest rev of this patch
> (https://reviews.apache.org/r/6377/) to be reviewed in hopes it can be
> committed and either way I will build the RC.
>
> Brock
>
> On Mon, Nov 5, 2012 at 9:47 AM, Nathaniel Auvil
> <[EMAIL PROTECTED]> wrote:
> > is there a target date for 1.3?
> >
> >
> > On Mon, Nov 5, 2012 at 10:34 AM, Brock Noland <[EMAIL PROTECTED]>
> wrote:
> >>
> >> There is actually an HTTP Source committed to the 1.3 branch:
> >>
> >>
> >>
> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c
> >>
> >> Otherwise I would shade or jarjar your HTTP Source jars so they don't
> >> conflict.
> >>
> >> Brock
> >>
> >> On Mon, Nov 5, 2012 at 9:31 AM, Nathaniel Auvil
> >> <[EMAIL PROTECTED]> wrote:
> >> > i am working with Flume 1.2.0 and creating my own HTTP source by
> >> > embedding
> >> > Tomcat. I am having several issues.  Has anyone ever gotten this to
> >> > work?
> >> > It looks like my latest issue is a servlet-api conflict due to an
> >> > ancient
> >> > Jetty 6 version on the Flume classpath
> >>
> >>
> >>
> >> --
> >> Apache MRUnit - Unit testing MapReduce -
> >> http://incubator.apache.org/mrunit/
> >
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>

Just curious, why doesn't the HTTP Source use Netty?

On Mon, Nov 5, 2012 at 1:09 PM, Nathaniel Auvil
<[EMAIL PROTECTED]>wrote:

> One thing i do not see with this HTTPSource is any way to customize the
> response.
>
>
> On Mon, Nov 5, 2012 at 10:58 AM, Brock Noland <[EMAIL PROTECTED]> wrote:
>
>> Very soon. I am going to wait for the latest rev of this patch
>> (https://reviews.apache.org/r/6377/) to be reviewed in hopes it can be
>> committed and either way I will build the RC.
>>
>> Brock
>>
>> On Mon, Nov 5, 2012 at 9:47 AM, Nathaniel Auvil
>> <[EMAIL PROTECTED]> wrote:
>> > is there a target date for 1.3?
>> >
>> >
>> > On Mon, Nov 5, 2012 at 10:34 AM, Brock Noland <[EMAIL PROTECTED]>
>> wrote:
>> >>
>> >> There is actually an HTTP Source committed to the 1.3 branch:
>> >>
>> >>
>> >>
>> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c
>> >>
>> >> Otherwise I would shade or jarjar your HTTP Source jars so they don't
>> >> conflict.
>> >>
>> >> Brock
>> >>
>> >> On Mon, Nov 5, 2012 at 9:31 AM, Nathaniel Auvil
>> >> <[EMAIL PROTECTED]> wrote:
>> >> > i am working with Flume 1.2.0 and creating my own HTTP source by
>> >> > embedding
>> >> > Tomcat. I am having several issues.  Has anyone ever gotten this to
>> >> > work?
>> >> > It looks like my latest issue is a servlet-api conflict due to an
>> >> > ancient
>> >> > Jetty 6 version on the Flume classpath
>> >>
>> >>
>> >>
>> >> --
>> >> Apache MRUnit - Unit testing MapReduce -
>> >> http://incubator.apache.org/mrunit/
>> >
>> >
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>>
>
>

No specific reason. I was familiar with Jetty and we already had a dependence on Jetty for the metrics stuff and I think some avro IPC stuff too, so I decided to just use the same.

Thanks,
Hari

--
Hari Shreedharan
On Monday, November 5, 2012 at 2:09 PM, Harish Mandala wrote:

> Just curious, why doesn't the HTTP Source use Netty?
>
> On Mon, Nov 5, 2012 at 1:09 PM, Nathaniel Auvil <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > One thing i do not see with this HTTPSource is any way to customize the response.
> >
> >
> > On Mon, Nov 5, 2012 at 10:58 AM, Brock Noland <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > Very soon. I am going to wait for the latest rev of this patch
> > > (https://reviews.apache.org/r/6377/) to be reviewed in hopes it can be
> > > committed and either way I will build the RC.
> > >
> > > Brock
> > >
> > > On Mon, Nov 5, 2012 at 9:47 AM, Nathaniel Auvil
> > > <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > > is there a target date for 1.3?
> > > >
> > > >
> > > > On Mon, Nov 5, 2012 at 10:34 AM, Brock Noland <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > >>
> > > >> There is actually an HTTP Source committed to the 1.3 branch:
> > > >>
> > > >>
> > > >> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c
> > > >>
> > > >> Otherwise I would shade or jarjar your HTTP Source jars so they don't
> > > >> conflict.
> > > >>
> > > >> Brock
> > > >>
> > > >> On Mon, Nov 5, 2012 at 9:31 AM, Nathaniel Auvil
> > > >> <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > >> > i am working with Flume 1.2.0 and creating my own HTTP source by
> > > >> > embedding
> > > >> > Tomcat. I am having several issues.  Has anyone ever gotten this to
> > > >> > work?
> > > >> > It looks like my latest issue is a servlet-api conflict due to an
> > > >> > ancient
> > > >> > Jetty 6 version on the Flume classpath
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Apache MRUnit - Unit testing MapReduce -
> > > >> http://incubator.apache.org/mrunit/
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
> >
>

Hi Nathaniel,

When I wrote this (and while it was being reviewed), we concluded it was not a good idea to expose the response object to the custom handler which can be plugged in because the handler can flush the buffer of the response object (which also can be flushed just because it is full) and then we could hit some exception (like channel being full) causing inconsistencies in the response. This source is not meant to be used for HTTP based RPC etc, this is to be used to send the data to the source and then wait for success or failure - on success send next event, on failure - back off for a while and retry (or similar logic).

For RPC etc, it might be best to write a source specific to your use-case. In case of this source bundled with Flume, we had to make it general enough to be useful but cautious enough to not make it confuse applications sending the data into Flume.

Hope this helps.

Thanks,
Hari

--
Hari Shreedharan
On Monday, November 5, 2012 at 10:09 AM, Nathaniel Auvil wrote:

> One thing i do not see with this HTTPSource is any way to customize the response.
>
>
> On Mon, Nov 5, 2012 at 10:58 AM, Brock Noland <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > Very soon. I am going to wait for the latest rev of this patch
> > (https://reviews.apache.org/r/6377/) to be reviewed in hopes it can be
> > committed and either way I will build the RC.
> >
> > Brock
> >
> > On Mon, Nov 5, 2012 at 9:47 AM, Nathaniel Auvil
> > <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > is there a target date for 1.3?
> > >
> > >
> > > On Mon, Nov 5, 2012 at 10:34 AM, Brock Noland <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > >>
> > >> There is actually an HTTP Source committed to the 1.3 branch:
> > >>
> > >>
> > >> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c
> > >>
> > >> Otherwise I would shade or jarjar your HTTP Source jars so they don't
> > >> conflict.
> > >>
> > >> Brock
> > >>
> > >> On Mon, Nov 5, 2012 at 9:31 AM, Nathaniel Auvil
> > >> <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > >> > i am working with Flume 1.2.0 and creating my own HTTP source by
> > >> > embedding
> > >> > Tomcat. I am having several issues.  Has anyone ever gotten this to
> > >> > work?
> > >> > It looks like my latest issue is a servlet-api conflict due to an
> > >> > ancient
> > >> > Jetty 6 version on the Flume classpath
> > >>
> > >>
> > >>
> > >> --
> > >> Apache MRUnit - Unit testing MapReduce -
> > >> http://incubator.apache.org/mrunit/
> > >
> > >
> >
> >
> >
> > --
> > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
>