Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> sqoop-export with sequence files doesn't work.


Copy link to this message
-
Re: sqoop-export with sequence files doesn't work.
Hi Deepak,
I would advise you to ask CDH specific questions on cdh-user [1] mailing lists as you are much more likely to get answer there.

For your convenience the Sqoop HCatalog integration will be part of upcoming CDH 4.4.0.

Jarcec

Links:
1: https://groups.google.com/a/cloudera.org/forum/#!forum/cdh-user

On Fri, Aug 16, 2013 at 11:04:34AM -0700, Venkat Ranganathan wrote:
> i don't know about CDH distro - may be it does.      You need Sqoop 1.4.4
> along with HCatalog installation
>
> Thanks
>
> Venkat
>
>
> On Fri, Aug 16, 2013 at 10:28 AM, Deepak Konidena <[EMAIL PROTECTED]>wrote:
>
> > @Krishna Rao - Sequence file provides us the compression we need.
> > Converting the data into a non-sequence file is not an option since it
> > won't be scale.
> >
> > @Venkat - The version of sqoop I am using is 1.4.3. Does HCatalog have to
> > be installed separately or does it come as part of CDH4.3 ?
> >
> >
> > -Deepak
> >
> >
> >
> > On Fri, Aug 16, 2013 at 10:14 AM, Venkat Ranganathan <
> > [EMAIL PROTECTED]> wrote:
> >
> >> The HCatalog integration handles tables with sequence files - It is part
> >> of 1.4.4 - It handles both exports and imports
> >>
> >> Venkat
> >>
> >>
> >> On Fri, Aug 16, 2013 at 9:32 AM, Krishna Rao <[EMAIL PROTECTED]>wrote:
> >>
> >>> I've run into this problem as well. I ended up copying the table into a
> >>> non-sequenceFile table just so I could sqoop it out (something along the
> >>> lines of CREATE TABLE nonSeqTbl LIKE seqTbl; INSERT OVERWRITE nonSeqTbl
> >>> SELECT * FROM seqTbl;).
> >>>
> >>> Is there a plan to allow sqoop-exporting of sequence file tables?
> >>>
> >>> Krishna
> >>>
> >>>
> >>> On 16 August 2013 17:26, Abraham Elmahrek <[EMAIL PROTECTED]> wrote:
> >>>
> >>>> Ah I believe you're correct. Was this data imported with Sqoop? If so,
> >>>> does the table you're exporting to differ from the table you imported from?
> >>>>
> >>>>
> >>>> On Thu, Aug 15, 2013 at 11:38 PM, Deepak Konidena <[EMAIL PROTECTED]
> >>>> > wrote:
> >>>>
> >>>>> Does sqoop-export support --as-sequence option? I know sqoop-import
> >>>>> does.
> >>>>>
> >>>>>
> >>>>> -Deepak
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Aug 15, 2013 at 11:34 PM, Abraham Elmahrek <[EMAIL PROTECTED]>wrote:
> >>>>>
> >>>>>> Hey There,
> >>>>>>
> >>>>>> I believe you're missing the --as-sequence directive!
> >>>>>>
> >>>>>> -Abe
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Aug 15, 2013 at 7:16 PM, Deepak Konidena <
> >>>>>> [EMAIL PROTECTED]> wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I have a sequence file with with both (key,value) as
> >>>>>>> org.apache.hadoop.io.Text
> >>>>>>>
> >>>>>>> I am trying to export the data into a mysql table with (key,value)
> >>>>>>> mapped to (varchar, blob) since the value is pretty big. and I get the
> >>>>>>> following error:
> >>>>>>>
> >>>>>>> (command) - sqoop export -m "1" -connect
> >>>>>>> "jdbc:mysql://<host>:3306/database" --username "sqoop" --password
> >>>>>>> "sqooppwd" --table "tablename"  --export-dir "/path/to/sequencefile"
> >>>>>>> --verbose
> >>>>>>>
> >>>>>>> java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be
> >>>>>>> cast to org.apache.hadoop.io.LongWritable
> >>>>>>>     at
> >>>>>>> org.apache.sqoop.mapreduce.CombineShimRecordReader.getCurrentKey(CombineShimRecordReader.java:95)
> >>>>>>>     at
> >>>>>>> org.apache.sqoop.mapreduce.CombineShimRecordReader.getCurrentKey(CombineShimRecordReader.java:38)
> >>>>>>>     at
> >>>>>>> org.apache.sqoop.mapreduce.CombineFileRecordReader.getCurrentKey(CombineFileRecordReader.java:79)
> >>>>>>>     at
> >>>>>>> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getCurrentKey(MapTask.java:461)
> >>>>>>>     at
> >>>>>>> org.apache.hadoop.mapreduce.task.MapContextImpl.getCurrentKey(MapContextImpl.java:66)
> >>>>>>>     at
> >>>>>>> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.getCurrentKey(WrappedMapper.java:75)
> >>>>>>>     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> >>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB