Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop, mail # user - Export SequenceFile compressed Hive table


+
Krishna Rao 2012-12-20, 10:37
Copy link to this message
-
Re: Export SequenceFile compressed Hive table
Chalcy 2012-12-20, 11:34
Hi Krishna,

No.  Sqoop import into hive is not possible yet.

We use sqoop import snappy compressed into hive and then create another
same schema sequence file output hive table and do a insert overwrite from
non sequence to sequence table.  This works very well except there is
another overhead.

For partitioned tables, the next select has to have all the fields
specified.

Hope this helps,
Chalcy

On Thu, Dec 20, 2012 at 5:37 AM, Krishna Rao <[EMAIL PROTECTED]> wrote:

> Hi all,
>
> Is it possible to export a SequenceFile compressed Hive table using Sqoop?
>
> It's absence from the docs, and this<https://issues.cloudera.org/browse/SQOOP-93>,
> suggest I can't.
>
> And attempting it results in:
> ***
> 12/12/20 10:15:13 INFO mapreduce.Job: Task Id :
> attempt_1354275827266_5834_m_000000_2, Status : FAILED
> Error: java.lang.ClassCastException: org.apache.hadoop.io.BytesWritable
> cannot be cast to org.apache.hadoop.io.LongWritable
>         at
> org.apache.sqoop.mapreduce.CombineShimRecordReader.getCurrentKey(CombineShimRecordReader.java:95)
>         at
> org.apache.sqoop.mapreduce.CombineShimRecordReader.getCurrentKey(CombineShimRecordReader.java:38)
>         at
> org.apache.sqoop.mapreduce.CombineFileRecordReader.getCurrentKey(CombineFileRecordReader.java:77)
>         at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getCurrentKey(MapTask.java:470)
>         at
> org.apache.hadoop.mapreduce.task.MapContextImpl.getCurrentKey(MapContextImpl.java:70)
>         at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.getCurrentKey(WrappedMapper.java:81)
>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>         at
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:183)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
>
> ***
>
> Cheers,
>
> Krishna
>