Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume NG MySQL sink


Copy link to this message
-
Re: Flume NG MySQL sink
Amit,

That sounds like a reasonable idea. You should be careful about handling SQL related issues (like duplicate primary keys etc). You'd probably need to add a lot of the SQL related configuration and failure handling
Thanks,
Hari
On Saturday, October 12, 2013 at 8:48 AM, Amit Sela wrote:

> Well, like I mentioned, I'm still new to Flume so maybe I'm totally of course here, but how about an implementation similar to HBaseSink ? replace the list of Puts with a list of Strings of insert statements ? execute them in batches ?
>
>
> On Thu, Oct 10, 2013 at 7:32 PM, Steve Morin <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > I would be interested in knowing your use-case too
> >
> >
> > On Thu, Oct 10, 2013 at 8:47 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > Hi Amit,
> > > to my best knowledge, there is no MySQL sink that would store data from Flume in MySQL. Would you mind describing your use case? I would be interesting to hear how the MySQL fits into the use case. Flume is heavy distributed system for collecting and injesting data, that can very easily overload single MySQL server, so I'm wondering if you are planning to have some sort of sharding or perhaps a mysql cluster to sustain the load?
> > >
> > > Jarcec
> > >
> > > On Thu, Oct 10, 2013 at 09:10:34AM +0200, Amit Sela wrote:
> > > > Hi all,
> > > >
> > > > I'm new to Flume and I would like to know if anyone used it to aggregate
> > > > logs and process them into MySQL (JDBC) inserts ?
> > > >
> > > > Thanks.
> >
>

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB