Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> S3 Sink in FlumeNG Configuration?


Copy link to this message
-
S3 Sink in FlumeNG Configuration?
Hey there,

I know this is a really newbish question, but I'm hoping to get a little
assistance here so I'm not stuck guess-and-checking.

I'm trying to figure out how to configure FlumeNG (1.3.1), but I couldn't
figure out how to setup the hdfs sink to use the s3 implementations.

I'm keeping track of my progress on this gist I made:
https://gist.github.com/crowdmatt/5256881

>From what I've gathered, I should be using the hdfs type, which I'm setting
up as such:

agent.sinks = s3Sink
agent.sinks.s3Sink.type = hdfs
agent.sinks.s3Sink.channel = recoverableMemoryChannel

... but that's where I end up hitting my head against the wall.  I know I
should be specifying my s3 access key, secret, and bucket in this format:
s3n://ACCESS_KEY_ID:SECRET_ACCESS_KEY@my-hdfs/

However, I don't know where to specify that, or what dot notation to use.

Can anyone point me in the right direction?

Best,
Matt
--
Matthew Moore
Co-Founder & CTO, CrowdMob Inc.
Mobile: (650) 888-5962

Need to schedule a meeting?  Invite me via Google Calendar!
[EMAIL PROTECTED]
+
Brock Noland 2013-03-29, 14:49
+
Matthew Moore 2013-03-29, 15:47
+
Matthew Moore 2013-03-29, 19:15
+
Matthew Moore 2013-03-29, 19:42