Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume (NG?) with S3 Sink?


Copy link to this message
-
Re: Flume (NG?) with S3 Sink?
There's a project over at Github
https://github.com/Hobsons/hobsons-flumeng-s3-sink
haven't tried it myself though.

On Tue, Aug 28, 2012 at 5:29 PM, Julian Henry Alcala <[EMAIL PROTECTED]>wrote:

> I tried flume-OG with s3 and it was not reliable.  I am interested in what
> people are doing with flume-ng and s3 as well.
>
>
> On Tue, Aug 28, 2012 at 1:47 PM, igor Finkelshteyn <[EMAIL PROTECTED]>wrote:
>
>> Has no one needed to send logs through Flume NG to s3? This JIRA<https://issues.apache.org/jira/browse/FLUME-1015> makes
>> it sound like it's something that's already been implemented with
>> documentation, but there is 0 documentation anywhere (definitely not in the
>> user guide), and the JIRA was resolved months ago.
>>
>>
>> On Aug 27, 2012, at 3:39 PM, igor Finkelshteyn wrote:
>>
>> Hi Folks,
>> I'm completely new to flume, but was hoping to use it as a way to collect
>> logs on a couple web servers and funnel them to an s3 sink in Avro. I've
>> found a tutorial for doing this on Flume OG, but am thinking I should
>> probably just work with Flume NG for a new project if that's possible. Is
>> there a prebuilt s3 sink for Flume NG that I can use? Can I just use the
>> HDFS sink and provide some extra parameters to get it to work with s3? Or
>> should I just use Flume OG?I'd also be open to using something else
>> entirely that has better s3 support, like FluentD, if that makes more
>> sense. Any ideas?
>>
>> Thanks!
>> Igor
>>
>>
>>
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB