Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> DBStorage to LOAD data via a SQL query?


Copy link to this message
-
Re: DBStorage to LOAD data via a SQL query?
Hadoop does it with DBInputFormt, you could take a look at that.

On Sun, Apr 29, 2012 at 12:21 PM, Russell Jurney
<[EMAIL PROTECTED]>wrote:

> Mightn't it be easy to write the SELECT part?  Not sure if the JDBC stuff
> is convenient that way.
>
> On Sun, Apr 29, 2012 at 12:18 PM, Prashant Kommireddi
> <[EMAIL PROTECTED]>wrote:
>
> > Ah I see what you are saying. You are right, I missed the SELECT part
> > entirely.
> >
> > On Sun, Apr 29, 2012 at 12:09 PM, Russell Jurney
> > <[EMAIL PROTECTED]>wrote:
> >
> > > Prashant, it has an INSERT query, but no SELECT query.  It does not
> > > implement getNext(), so it looks like it is STORE only, not LOAD.  Am I
> > > mistaken?  I read the source, but it was late :)
> > >
> > > On Sun, Apr 29, 2012 at 12:04 PM, Prashant Kommireddi
> > > <[EMAIL PROTECTED]>wrote:
> > >
> > > > Russell,
> > > >
> > > > Looking at source code for DBStorage, seems like it does exactly
> that.
> > > Can
> > > > you try it out?
> > > >
> > > > public DBStorage(String driver, String jdbcURL, String user, String
> > pass,
> > > >      String insertQuery, String batchSize)
> > > >
> > > > Thanks,
> > > > Prashant
> > > >
> > > >
> > > >
> > > > On Sat, Apr 28, 2012 at 12:22 AM, Russell Jurney
> > > > <[EMAIL PROTECTED]>wrote:
> > > >
> > > > > Is it possible to use DBStorage to load data from MySQL by running
> a
> > > > > suppled SQL query? Something like:
> > > > >
> > > > > mydata = LOAD 'jdbc://localhost/enron' USING DBStorage('SELECT
> > > > foo.value1,
> > > > > bar.value2 FROM foo JOIN bar on foo.bar_id = bar.id');
> > > > >
> > > > >
> > > > > Even if I have to LOAD AS and specify a schema, that would be
> great.
> > > > >
> > > > > It is problematic that there are no docs for DBStorage. If someone
> > > clues
> > > > me
> > > > > in, I'll write it up :)
> > > > >
> > > > > --
> > > > > Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
> > > > > datasyndrome.com
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
> > > datasyndrome.com
> > >
> >
>
>
>
> --
> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
> datasyndrome.com
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB