Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Seeking Help configuring log4j for sqoop import into hive


+
Sunita Arvind 2013-11-11, 21:48
+
Sunita Arvind 2013-11-11, 23:01
Copy link to this message
-
Re: Seeking Help configuring log4j for sqoop import into hive
On 12 Nov 2013, at 0:01, Sunita Arvind wrote:

> Just in case this acts as a workaround for someone:
> The issue is resolved if I eliminate the "where" clause in the query
> (just
> keep "where $CONDITIONS"). So 2 workarounds I can think of now are:
> 1. Create views in Oracle and query without the where clause in the
> sqoop
> import command
> 2. Import everything in the table (not feasible in most cases)
>
> However, I still need to know how to get the exact stack trace.
>
> regards
> Sunita
>
>
> On Mon, Nov 11, 2013 at 1:48 PM, Sunita Arvind
> <[EMAIL PROTECTED]>wrote:
>
>> Hello,
>>
>> I am using sqoop to import data from oracle into hive. Below is my
>> SQL:
>>
>> nohup sqoop import --connect "jdbc:oracle:thin:@(DESCRIPTION =
>> (ADDRESS >> (PROTOCOL = TCP)(HOST = xxxxxxx)(PORT = xxxx)) (CONNECT_DATA =
>> (SERVER >> DEDICATED) (SERVICE_NAME = CDWQ.tms.toyota.com) (FAILOVER_MODE>> (TYPE=select) (METHOD=basic))))"  --username "xxxx"  --password
>> "xxxx"
>> --split-by employeeid --query  "SELECT e.employeeid,p.salary from
>> employee
>> e, payroll p
>> where e.employeeid =p.employeeid and $CONDITIONS"
>> --create-hive-table  --hive-table "EMPLOYEE" --hive-import
>> --target-dir
>> "/user/hive/warehouse/employee" --direct --verbose
>>
>>
>> Note: This is production data hence I cannot share the log file or
>> actual
>> query. Sorry for that.
>>
>> Similar query works for some tables and for this particular table,
>> there
>> is an exception as below:
>>
>> java.io.IOException: SQLException in nextKeyValue
>>      at
>> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:266)
>>      at
>> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:484)
>>      at
>> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76)
>>      at
>> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85)
>>      at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139)
>>      at
>> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
>>      at
>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:673)
>>      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
>>      at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>>      at java.security.AccessController.doPrivileged(Native Method)
>>      at javax.security.auth.Subject.doAs(Subject.java:396)
>>      at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>      at org.apache.hadoop.mapred.Child.main(Child.java:262)
This is usually the case when your PK (on which Sqoop will try to do the
split) isn't an integer.

my 2c.

David
+
Sunita Arvind 2013-11-12, 14:53
+
Jarek Jarcec Cecho 2013-11-12, 00:33
+
Sunita Arvind 2013-11-12, 13:29
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB