I've made a decent amount of progress, and now have the settings correct.
For completeness, the settings look like this:
agent.sinks.s3Sink.type = hdfs
You can see the full setup at this gist:
However, I've run into the following problem:
2013-03-29 19:05:28,954 (SinkRunner-PollingRunner-DefaultSinkProcessor)
Request Error. HEAD '/FlumeData.1364583927762.tmp' on Host '
mybucket.s3.amazonaws.com' @ 'Fri, 29 Mar 2013 19:05:28 GMT' --
ResponseCode: 404, ResponseStatus: Not Found, RequestId: 00864FE1DCD5AD95,
Does anyone have any pointers on how I can start debugging?
Co-Founder & CTO, CrowdMob Inc.
Mobile: (650) 888-5962
Need to schedule a meeting? Invite me via Google Calendar!
On Fri, Mar 29, 2013 at 8:47 AM, Matthew Moore <[EMAIL PROTECTED]> wrote:
> Thanks for the links to the Jiras. It seems like someone implemented
> an S3BufferedWriter which might be helpful in the future.
> However, I'm still not sure what to set the configuration (flume.conf) to
> use s3 as a sink? Has anyone done that?
> Matthew Moore
> Co-Founder & CTO, CrowdMob Inc.
> Mobile: (650) 888-5962
> Need to schedule a meeting? Invite me via Google Calendar!
> [EMAIL PROTECTED]
> On Fri, Mar 29, 2013 at 7:49 AM, Brock Noland <[EMAIL PROTECTED]> wrote:
>> Sorry, I don't know much about this, but here are two relevant JIRA's:
>> On Fri, Mar 29, 2013 at 9:44 AM, Matthew Moore <[EMAIL PROTECTED]> wrote:
>>> Hey there,
>>> I know this is a really newbish question, but I'm hoping to get a little
>>> assistance here so I'm not stuck guess-and-checking.
>>> I'm trying to figure out how to configure FlumeNG (1.3.1), but I
>>> couldn't figure out how to setup the hdfs sink to use the s3
>>> I'm keeping track of my progress on this gist I made:
>>> From what I've gathered, I should be using the hdfs type, which I'm
>>> setting up as such:
>>> agent.sinks = s3Sink
>>> agent.sinks.s3Sink.type = hdfs
>>> agent.sinks.s3Sink.channel = recoverableMemoryChannel
>>> ... but that's where I end up hitting my head against the wall. I know
>>> I should be specifying my s3 access key, secret, and bucket in this format:
>>> However, I don't know where to specify that, or what dot notation to use.
>>> Can anyone point me in the right direction?
>>> Matthew Moore
>>> Co-Founder & CTO, CrowdMob Inc.
>>> Mobile: (650) 888-5962
>>> Need to schedule a meeting? Invite me via Google Calendar!
>>> [EMAIL PROTECTED]
>> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org