Hi Folks,
I'm completely new to flume, but was hoping to use it as a way to collect logs on a couple web servers and funnel them to an s3 sink in Avro. I've found a tutorial for doing this on Flume OG, but am thinking I should probably just work with Flume NG for a new project if that's possible. Is there a prebuilt s3 sink for Flume NG that I can use? Can I just use the HDFS sink and provide some extra parameters to get it to work with s3? Or should I just use Flume OG?I'd also be open to using something else entirely that has better s3 support, like FluentD, if that makes more sense. Any ideas?

Thanks!
Igor

Has no one needed to send logs through Flume NG to s3? This JIRA makes it sound like it's something that's already been implemented with documentation, but there is 0 documentation anywhere (definitely not in the user guide), and the JIRA was resolved months ago.
On Aug 27, 2012, at 3:39 PM, igor Finkelshteyn wrote:

> Hi Folks,
> I'm completely new to flume, but was hoping to use it as a way to collect logs on a couple web servers and funnel them to an s3 sink in Avro. I've found a tutorial for doing this on Flume OG, but am thinking I should probably just work with Flume NG for a new project if that's possible. Is there a prebuilt s3 sink for Flume NG that I can use? Can I just use the HDFS sink and provide some extra parameters to get it to work with s3? Or should I just use Flume OG?I'd also be open to using something else entirely that has better s3 support, like FluentD, if that makes more sense. Any ideas?
>
> Thanks!
> Igor
>
>

I tried flume-OG with s3 and it was not reliable.  I am interested in what
people are doing with flume-ng and s3 as well.

On Tue, Aug 28, 2012 at 1:47 PM, igor Finkelshteyn <[EMAIL PROTECTED]>wrote:

> Has no one needed to send logs through Flume NG to s3? This JIRA<https://issues.apache.org/jira/browse/FLUME-1015> makes
> it sound like it's something that's already been implemented with
> documentation, but there is 0 documentation anywhere (definitely not in the
> user guide), and the JIRA was resolved months ago.
>
>
> On Aug 27, 2012, at 3:39 PM, igor Finkelshteyn wrote:
>
> Hi Folks,
> I'm completely new to flume, but was hoping to use it as a way to collect
> logs on a couple web servers and funnel them to an s3 sink in Avro. I've
> found a tutorial for doing this on Flume OG, but am thinking I should
> probably just work with Flume NG for a new project if that's possible. Is
> there a prebuilt s3 sink for Flume NG that I can use? Can I just use the
> HDFS sink and provide some extra parameters to get it to work with s3? Or
> should I just use Flume OG?I'd also be open to using something else
> entirely that has better s3 support, like FluentD, if that makes more
> sense. Any ideas?
>
> Thanks!
> Igor
>
>
>
>

FLUME-1015 is saying we could use the Hadoop S3 FileSystem
abstraction. If it doesn't work, it's likely a hadoop problem.
FLUME-951 is for implementing a direct to S3 sink.

On Tue, Aug 28, 2012 at 4:29 PM, Julian Henry Alcala <[EMAIL PROTECTED]> wrote:
> I tried flume-OG with s3 and it was not reliable.  I am interested in what
> people are doing with flume-ng and s3 as well.
>
>
> On Tue, Aug 28, 2012 at 1:47 PM, igor Finkelshteyn <[EMAIL PROTECTED]>
> wrote:
>>
>> Has no one needed to send logs through Flume NG to s3? This JIRA makes it
>> sound like it's something that's already been implemented with
>> documentation, but there is 0 documentation anywhere (definitely not in the
>> user guide), and the JIRA was resolved months ago.
>>
>>
>> On Aug 27, 2012, at 3:39 PM, igor Finkelshteyn wrote:
>>
>> Hi Folks,
>> I'm completely new to flume, but was hoping to use it as a way to collect
>> logs on a couple web servers and funnel them to an s3 sink in Avro. I've
>> found a tutorial for doing this on Flume OG, but am thinking I should
>> probably just work with Flume NG for a new project if that's possible. Is
>> there a prebuilt s3 sink for Flume NG that I can use? Can I just use the
>> HDFS sink and provide some extra parameters to get it to work with s3? Or
>> should I just use Flume OG?I'd also be open to using something else entirely
>> that has better s3 support, like FluentD, if that makes more sense. Any
>> ideas?
>>
>> Thanks!
>> Igor
>>
>>
>>
>

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

There's a project over at Github
https://github.com/Hobsons/hobsons-flumeng-s3-sink
haven't tried it myself though.

On Tue, Aug 28, 2012 at 5:29 PM, Julian Henry Alcala <[EMAIL PROTECTED]>wrote:

> I tried flume-OG with s3 and it was not reliable.  I am interested in what
> people are doing with flume-ng and s3 as well.
>
>
> On Tue, Aug 28, 2012 at 1:47 PM, igor Finkelshteyn <[EMAIL PROTECTED]>wrote:
>
>> Has no one needed to send logs through Flume NG to s3? This JIRA<https://issues.apache.org/jira/browse/FLUME-1015> makes
>> it sound like it's something that's already been implemented with
>> documentation, but there is 0 documentation anywhere (definitely not in the
>> user guide), and the JIRA was resolved months ago.
>>
>>
>> On Aug 27, 2012, at 3:39 PM, igor Finkelshteyn wrote:
>>
>> Hi Folks,
>> I'm completely new to flume, but was hoping to use it as a way to collect
>> logs on a couple web servers and funnel them to an s3 sink in Avro. I've
>> found a tutorial for doing this on Flume OG, but am thinking I should
>> probably just work with Flume NG for a new project if that's possible. Is
>> there a prebuilt s3 sink for Flume NG that I can use? Can I just use the
>> HDFS sink and provide some extra parameters to get it to work with s3? Or
>> should I just use Flume OG?I'd also be open to using something else
>> entirely that has better s3 support, like FluentD, if that makes more
>> sense. Any ideas?
>>
>> Thanks!
>> Igor
>>
>>
>>
>>
>