Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Seeking Help configuring log4j for sqoop import into hive


Copy link to this message
-
Re: Seeking Help configuring log4j for sqoop import into hive
Jarek Jarcec Cecho 2013-11-12, 00:33
Hi Sunita,
Sqoop specific questions are better asked on Sqoop user mailing list [EMAIL PROTECTED]. You can find instructions how to subscribe to that at [1].

I would suggest to take a look into the failed map task log as that log usually contain entire exception including all the chained exceptions.

Jarcec

Links:
1: http://sqoop.apache.org/mail-lists.html

On Mon, Nov 11, 2013 at 03:01:22PM -0800, Sunita Arvind wrote:
> Just in case this acts as a workaround for someone:
> The issue is resolved if I eliminate the "where" clause in the query (just
> keep "where $CONDITIONS"). So 2 workarounds I can think of now are:
> 1. Create views in Oracle and query without the where clause in the sqoop
> import command
> 2. Import everything in the table (not feasible in most cases)
>
> However, I still need to know how to get the exact stack trace.
>
> regards
> Sunita
>
>
> On Mon, Nov 11, 2013 at 1:48 PM, Sunita Arvind <[EMAIL PROTECTED]>wrote:
>
> > Hello,
> >
> > I am using sqoop to import data from oracle into hive. Below is my SQL:
> >
> > nohup sqoop import --connect "jdbc:oracle:thin:@(DESCRIPTION = (ADDRESS > > (PROTOCOL = TCP)(HOST = xxxxxxx)(PORT = xxxx)) (CONNECT_DATA = (SERVER > > DEDICATED) (SERVICE_NAME = CDWQ.tms.toyota.com) (FAILOVER_MODE> > (TYPE=select) (METHOD=basic))))"  --username "xxxx"  --password "xxxx"
> > --split-by employeeid --query  "SELECT e.employeeid,p.salary from employee
> > e, payroll p
> > where e.employeeid =p.employeeid and $CONDITIONS"
> > --create-hive-table  --hive-table "EMPLOYEE" --hive-import --target-dir
> > "/user/hive/warehouse/employee" --direct --verbose
> >
> >
> > Note: This is production data hence I cannot share the log file or actual
> > query. Sorry for that.
> >
> > Similar query works for some tables and for this particular table, there
> > is an exception as below:
> >
> > java.io.IOException: SQLException in nextKeyValue
> >         at
> > org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:266)
> >         at
> > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:484)
> >         at
> > org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76)
> >         at
> > org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85)
> >         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139)
> >         at
> > org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
> >         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:673)
> >         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
> >         at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> >         at java.security.AccessController.doPrivileged(Native Method)
> >         at javax.security.auth.Subject.doAs(Subject.java:396)
> >         at
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> >         at org.apache.hadoop.mapred.Child.main(Child.java:262)
> > Caused by: java.sql
> > attempt_201311071517_0011_m_000003_0: log4j:WARN No appenders could be
> > found for logger (org.apache.hadoop.hdfs.DFSClient).
> > attempt_201311071517_0011_m_000003_0: log4j:WARN Please initialize the
> > log4j system properly.
> > attempt_201311071517_0011_m_000003_0: log4j:WARN See
> > http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> > 13/11/11 11:35:20 INFO mapred.JobClient: Task Id :
> > attempt_201311071517_0011_m_000000_0, Status : FAILED
> >
> >
> > I eye-balled to see date format issues, which is typically the root cause
> > for such issues, as per forums. But that does not seem to be the case here
> > (I could be wrong). I also added the "-direct" option as suggested by some
> > posts and that did not help either.
> >
> > The actual exception after the "caused by" is missing, which makes me
> > believe that sqoop is trying to redirect the output to some log file and it
> > does not find the necessary configurations. Hence it is not dumping the