Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> sqoop-export with sequence files doesn't work.


+
Deepak Konidena 2013-08-16, 02:16
+
Abraham Elmahrek 2013-08-16, 06:34
+
Deepak Konidena 2013-08-16, 06:38
+
Abraham Elmahrek 2013-08-16, 16:26
+
Krishna Rao 2013-08-16, 16:32
+
Venkat Ranganathan 2013-08-16, 17:14
+
Deepak Konidena 2013-08-16, 17:28
+
Venkat Ranganathan 2013-08-16, 18:04
Copy link to this message
-
Re: sqoop-export with sequence files doesn't work.
Hi Deepak,
I would advise you to ask CDH specific questions on cdh-user [1] mailing lists as you are much more likely to get answer there.

For your convenience the Sqoop HCatalog integration will be part of upcoming CDH 4.4.0.

Jarcec

Links:
1: https://groups.google.com/a/cloudera.org/forum/#!forum/cdh-user

On Fri, Aug 16, 2013 at 11:04:34AM -0700, Venkat Ranganathan wrote:
> i don't know about CDH distro - may be it does.      You need Sqoop 1.4.4
> along with HCatalog installation
>
> Thanks
>
> Venkat
>
>
> On Fri, Aug 16, 2013 at 10:28 AM, Deepak Konidena <[EMAIL PROTECTED]>wrote:
>
> > @Krishna Rao - Sequence file provides us the compression we need.
> > Converting the data into a non-sequence file is not an option since it
> > won't be scale.
> >
> > @Venkat - The version of sqoop I am using is 1.4.3. Does HCatalog have to
> > be installed separately or does it come as part of CDH4.3 ?
> >
> >
> > -Deepak
> >
> >
> >
> > On Fri, Aug 16, 2013 at 10:14 AM, Venkat Ranganathan <
> > [EMAIL PROTECTED]> wrote:
> >
> >> The HCatalog integration handles tables with sequence files - It is part
> >> of 1.4.4 - It handles both exports and imports
> >>
> >> Venkat
> >>
> >>
> >> On Fri, Aug 16, 2013 at 9:32 AM, Krishna Rao <[EMAIL PROTECTED]>wrote:
> >>
> >>> I've run into this problem as well. I ended up copying the table into a
> >>> non-sequenceFile table just so I could sqoop it out (something along the
> >>> lines of CREATE TABLE nonSeqTbl LIKE seqTbl; INSERT OVERWRITE nonSeqTbl
> >>> SELECT * FROM seqTbl;).
> >>>
> >>> Is there a plan to allow sqoop-exporting of sequence file tables?
> >>>
> >>> Krishna
> >>>
> >>>
> >>> On 16 August 2013 17:26, Abraham Elmahrek <[EMAIL PROTECTED]> wrote:
> >>>
> >>>> Ah I believe you're correct. Was this data imported with Sqoop? If so,
> >>>> does the table you're exporting to differ from the table you imported from?
> >>>>
> >>>>
> >>>> On Thu, Aug 15, 2013 at 11:38 PM, Deepak Konidena <[EMAIL PROTECTED]
> >>>> > wrote:
> >>>>
> >>>>> Does sqoop-export support --as-sequence option? I know sqoop-import
> >>>>> does.
> >>>>>
> >>>>>
> >>>>> -Deepak
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Aug 15, 2013 at 11:34 PM, Abraham Elmahrek <[EMAIL PROTECTED]>wrote:
> >>>>>
> >>>>>> Hey There,
> >>>>>>
> >>>>>> I believe you're missing the --as-sequence directive!
> >>>>>>
> >>>>>> -Abe
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Aug 15, 2013 at 7:16 PM, Deepak Konidena <
> >>>>>> [EMAIL PROTECTED]> wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I have a sequence file with with both (key,value) as
> >>>>>>> org.apache.hadoop.io.Text
> >>>>>>>
> >>>>>>> I am trying to export the data into a mysql table with (key,value)
> >>>>>>> mapped to (varchar, blob) since the value is pretty big. and I get the
> >>>>>>> following error:
> >>>>>>>
> >>>>>>> (command) - sqoop export -m "1" -connect
> >>>>>>> "jdbc:mysql://<host>:3306/database" --username "sqoop" --password
> >>>>>>> "sqooppwd" --table "tablename"  --export-dir "/path/to/sequencefile"
> >>>>>>> --verbose
> >>>>>>>
> >>>>>>> java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be
> >>>>>>> cast to org.apache.hadoop.io.LongWritable
> >>>>>>>     at
> >>>>>>> org.apache.sqoop.mapreduce.CombineShimRecordReader.getCurrentKey(CombineShimRecordReader.java:95)
> >>>>>>>     at
> >>>>>>> org.apache.sqoop.mapreduce.CombineShimRecordReader.getCurrentKey(CombineShimRecordReader.java:38)
> >>>>>>>     at
> >>>>>>> org.apache.sqoop.mapreduce.CombineFileRecordReader.getCurrentKey(CombineFileRecordReader.java:79)
> >>>>>>>     at
> >>>>>>> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getCurrentKey(MapTask.java:461)
> >>>>>>>     at
> >>>>>>> org.apache.hadoop.mapreduce.task.MapContextImpl.getCurrentKey(MapContextImpl.java:66)
> >>>>>>>     at
> >>>>>>> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.getCurrentKey(WrappedMapper.java:75)
> >>>>>>>     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> >>