Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> S3 Sink in FlumeNG Configuration?


Copy link to this message
-
Re: S3 Sink in FlumeNG Configuration?
Sorry, I don't know much about this, but here are two relevant JIRA's:

https://issues.apache.org/jira/browse/FLUME-1228
https://issues.apache.org/jira/browse/FLUME-951
On Fri, Mar 29, 2013 at 9:44 AM, Matthew Moore <[EMAIL PROTECTED]> wrote:

> Hey there,
>
> I know this is a really newbish question, but I'm hoping to get a little
> assistance here so I'm not stuck guess-and-checking.
>
> I'm trying to figure out how to configure FlumeNG (1.3.1), but I couldn't
> figure out how to setup the hdfs sink to use the s3 implementations.
>
> I'm keeping track of my progress on this gist I made:
> https://gist.github.com/crowdmatt/5256881
>
> From what I've gathered, I should be using the hdfs type, which I'm
> setting up as such:
>
> agent.sinks = s3Sink
> agent.sinks.s3Sink.type = hdfs
> agent.sinks.s3Sink.channel = recoverableMemoryChannel
>
> ... but that's where I end up hitting my head against the wall.  I know I
> should be specifying my s3 access key, secret, and bucket in this format:
> s3n://ACCESS_KEY_ID:SECRET_ACCESS_KEY@my-hdfs/
>
> However, I don't know where to specify that, or what dot notation to use.
>
> Can anyone point me in the right direction?
>
> Best,
> Matt
> --
> Matthew Moore
> Co-Founder & CTO, CrowdMob Inc.
> Mobile: (650) 888-5962
>
> Need to schedule a meeting?  Invite me via Google Calendar!
> [EMAIL PROTECTED]
>

--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org