Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - DBStorage to LOAD data via a SQL query?


Copy link to this message
-
Re: DBStorage to LOAD data via a SQL query?
Prashant Kommireddi 2012-04-29, 19:53
Hadoop does it with DBInputFormt, you could take a look at that.

On Sun, Apr 29, 2012 at 12:21 PM, Russell Jurney
<[EMAIL PROTECTED]>wrote:

> Mightn't it be easy to write the SELECT part?  Not sure if the JDBC stuff
> is convenient that way.
>
> On Sun, Apr 29, 2012 at 12:18 PM, Prashant Kommireddi
> <[EMAIL PROTECTED]>wrote:
>
> > Ah I see what you are saying. You are right, I missed the SELECT part
> > entirely.
> >
> > On Sun, Apr 29, 2012 at 12:09 PM, Russell Jurney
> > <[EMAIL PROTECTED]>wrote:
> >
> > > Prashant, it has an INSERT query, but no SELECT query.  It does not
> > > implement getNext(), so it looks like it is STORE only, not LOAD.  Am I
> > > mistaken?  I read the source, but it was late :)
> > >
> > > On Sun, Apr 29, 2012 at 12:04 PM, Prashant Kommireddi
> > > <[EMAIL PROTECTED]>wrote:
> > >
> > > > Russell,
> > > >
> > > > Looking at source code for DBStorage, seems like it does exactly
> that.
> > > Can
> > > > you try it out?
> > > >
> > > > public DBStorage(String driver, String jdbcURL, String user, String
> > pass,
> > > >      String insertQuery, String batchSize)
> > > >
> > > > Thanks,
> > > > Prashant
> > > >
> > > >
> > > >
> > > > On Sat, Apr 28, 2012 at 12:22 AM, Russell Jurney
> > > > <[EMAIL PROTECTED]>wrote:
> > > >
> > > > > Is it possible to use DBStorage to load data from MySQL by running
> a
> > > > > suppled SQL query? Something like:
> > > > >
> > > > > mydata = LOAD 'jdbc://localhost/enron' USING DBStorage('SELECT
> > > > foo.value1,
> > > > > bar.value2 FROM foo JOIN bar on foo.bar_id = bar.id');
> > > > >
> > > > >
> > > > > Even if I have to LOAD AS and specify a schema, that would be
> great.
> > > > >
> > > > > It is problematic that there are no docs for DBStorage. If someone
> > > clues
> > > > me
> > > > > in, I'll write it up :)
> > > > >
> > > > > --
> > > > > Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
> > > > > datasyndrome.com
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
> > > datasyndrome.com
> > >
> >
>
>
>
> --
> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
> datasyndrome.com
>