Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> Mysql Direct export of Avro files


Copy link to this message
-
Mysql Direct export of Avro files
Thanks a lot for replying, Jarek. (Dont know how i missed adding a subject
- my bad)
Have opened a feature request -
SQOOP-1146<https://issues.apache.org/jira/browse/SQOOP-1146>

Thanks,
Rama.
On Fri, Jul 19, 2013 at 8:29 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:

> Hi Ramakrishna,
> indeed your investigation is correct, the MySQL direct import/export do
> not supports file formats other than text. Please do not hesitate and open
> a feature request for it [1].
>
> All emails sent to apache mailing are publicly archived so that people
> looking having questions can easily search the list if the question was
> already answered. As the subject field significantly helps during search,
> please do fill this field next time.
>
> Jarcec
>
> Links:
> 1: https://issues.apache.org/jira/browse/SQOOP
>
> On Fri, Jul 19, 2013 at 05:12:34PM +0530, Ramakrishna Nalam wrote:
> > Update: Did some more digging into the logic of sqoop, realized the avro
> > check happens only in JDBCExportJob, and DirectMySQLManager uses
> > MysqlExportJob that does not extend JDBCExportJob.
> > Guessing avro is supported only for record based inputs and not bulk
> import
> > of mysql.
> >
> > The mapper itself seems to iterate over the rows, though. Can anyone help
> > me out with why is the avro format not supported by MysqlExportJob?
> >
> > Regards,
> > Rama.
> >
> >
> >
> > On Fri, Jul 19, 2013 at 9:27 AM, Ramakrishna Nalam
> > <[EMAIL PROTECTED]>wrote:
> >
> > >
> > > (Pls ignore the --direct pasted twice, I'm using it only once, the
> problem
> > > still remains)
> > >
> > >
> > > On Fri, Jul 19, 2013 at 9:25 AM, Ramakrishna Nalam <
> [EMAIL PROTECTED]
> > > > wrote:
> > >
> > >> Hi,
> > >>
> > >> I'm trying to export a set of avro files to mysql using the
> > >> DirectMysqlManager, using the sqoop export command as follows:
> > >>
> > >> sqoop export  --verbose  --connect jdbc:mysql://<mysqluri>/<dbname>
> > >> --direct   --username <username> --password <password> --direct
> > >> --export-dir <dir with avro files> --table <tablename>
> > >>
> > >>
> > >> On running this command, I see that the job's
> *mapred.inputformat.class* as
> > >> '*org.apache.sqoop.mapreduce.ExportInputFormat'* wheras I was
> expecting
> > >> it to set it to '*org.apache.sqoop.mapreduce.AvroInputFormat**'*
> > >>
> > >> I have also checked that*
> org.apache.sqoop.mapreduce.ExportJobBase.getFileType(conf
> > >> , path ) *on the same path returns *AVRO_DATA_FILE*
> > >>
> > >> Please let me know if I'm missing something here?
> > >>
> > >> TIA,
> > >> Rama.
> > >>
> > >
> > >
>
+
Jarek Jarcec Cecho 2013-07-20, 02:13
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB