-Re: Flume NG MySQL sink
Hari Shreedharan 2013-10-12, 20:17
That sounds like a reasonable idea. You should be careful about handling SQL related issues (like duplicate primary keys etc). You'd probably need to add a lot of the SQL related configuration and failure handling
On Saturday, October 12, 2013 at 8:48 AM, Amit Sela wrote:
> Well, like I mentioned, I'm still new to Flume so maybe I'm totally of course here, but how about an implementation similar to HBaseSink ? replace the list of Puts with a list of Strings of insert statements ? execute them in batches ?
> On Thu, Oct 10, 2013 at 7:32 PM, Steve Morin <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > I would be interested in knowing your use-case too
> > On Thu, Oct 10, 2013 at 8:47 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > Hi Amit,
> > > to my best knowledge, there is no MySQL sink that would store data from Flume in MySQL. Would you mind describing your use case? I would be interesting to hear how the MySQL fits into the use case. Flume is heavy distributed system for collecting and injesting data, that can very easily overload single MySQL server, so I'm wondering if you are planning to have some sort of sharding or perhaps a mysql cluster to sustain the load?
> > >
> > > Jarcec
> > >
> > > On Thu, Oct 10, 2013 at 09:10:34AM +0200, Amit Sela wrote:
> > > > Hi all,
> > > >
> > > > I'm new to Flume and I would like to know if anyone used it to aggregate
> > > > logs and process them into MySQL (JDBC) inserts ?
> > > >
> > > > Thanks.