Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive, mail # user - Seeking Help configuring log4j for sqoop import into hive


+
Sunita Arvind 2013-11-11, 21:48
+
Sunita Arvind 2013-11-11, 23:01
Copy link to this message
-
Re: Seeking Help configuring log4j for sqoop import into hive
David Morel 2013-11-12, 00:52
On 12 Nov 2013, at 0:01, Sunita Arvind wrote:

> Just in case this acts as a workaround for someone:
> The issue is resolved if I eliminate the "where" clause in the query
> (just
> keep "where $CONDITIONS"). So 2 workarounds I can think of now are:
> 1. Create views in Oracle and query without the where clause in the
> sqoop
> import command
> 2. Import everything in the table (not feasible in most cases)
>
> However, I still need to know how to get the exact stack trace.
>
> regards
> Sunita
>
>
> On Mon, Nov 11, 2013 at 1:48 PM, Sunita Arvind
> <[EMAIL PROTECTED]>wrote:
>
>> Hello,
>>
>> I am using sqoop to import data from oracle into hive. Below is my
>> SQL:
>>
>> nohup sqoop import --connect "jdbc:oracle:thin:@(DESCRIPTION =
>> (ADDRESS >> (PROTOCOL = TCP)(HOST = xxxxxxx)(PORT = xxxx)) (CONNECT_DATA =
>> (SERVER >> DEDICATED) (SERVICE_NAME = CDWQ.tms.toyota.com) (FAILOVER_MODE>> (TYPE=select) (METHOD=basic))))"  --username "xxxx"  --password
>> "xxxx"
>> --split-by employeeid --query  "SELECT e.employeeid,p.salary from
>> employee
>> e, payroll p
>> where e.employeeid =p.employeeid and $CONDITIONS"
>> --create-hive-table  --hive-table "EMPLOYEE" --hive-import
>> --target-dir
>> "/user/hive/warehouse/employee" --direct --verbose
>>
>>
>> Note: This is production data hence I cannot share the log file or
>> actual
>> query. Sorry for that.
>>
>> Similar query works for some tables and for this particular table,
>> there
>> is an exception as below:
>>
>> java.io.IOException: SQLException in nextKeyValue
>>      at
>> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:266)
>>      at
>> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:484)
>>      at
>> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76)
>>      at
>> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85)
>>      at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139)
>>      at
>> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
>>      at
>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:673)
>>      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
>>      at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>>      at java.security.AccessController.doPrivileged(Native Method)
>>      at javax.security.auth.Subject.doAs(Subject.java:396)
>>      at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>      at org.apache.hadoop.mapred.Child.main(Child.java:262)
This is usually the case when your PK (on which Sqoop will try to do the
split) isn't an integer.

my 2c.

David
+
Sunita Arvind 2013-11-12, 14:53
+
Jarek Jarcec Cecho 2013-11-12, 00:33
+
Sunita Arvind 2013-11-12, 13:29