Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> S3 Sink in FlumeNG Configuration?


Copy link to this message
-
S3 Sink in FlumeNG Configuration?
Hey there,

I know this is a really newbish question, but I'm hoping to get a little
assistance here so I'm not stuck guess-and-checking.

I'm trying to figure out how to configure FlumeNG (1.3.1), but I couldn't
figure out how to setup the hdfs sink to use the s3 implementations.

I'm keeping track of my progress on this gist I made:
https://gist.github.com/crowdmatt/5256881

>From what I've gathered, I should be using the hdfs type, which I'm setting
up as such:

agent.sinks = s3Sink
agent.sinks.s3Sink.type = hdfs
agent.sinks.s3Sink.channel = recoverableMemoryChannel

... but that's where I end up hitting my head against the wall.  I know I
should be specifying my s3 access key, secret, and bucket in this format:
s3n://ACCESS_KEY_ID:SECRET_ACCESS_KEY@my-hdfs/

However, I don't know where to specify that, or what dot notation to use.

Can anyone point me in the right direction?

Best,
Matt
--
Matthew Moore
Co-Founder & CTO, CrowdMob Inc.
Mobile: (650) 888-5962

Need to schedule a meeting?  Invite me via Google Calendar!
[EMAIL PROTECTED]
+
Brock Noland 2013-03-29, 14:49
+
Matthew Moore 2013-03-29, 15:47
+
Matthew Moore 2013-03-29, 19:15
+
Matthew Moore 2013-03-29, 19:42
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB