Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> Tips on finding the file that is causing sqoop to choke


Copy link to this message
-
Tips on finding the file that is causing sqoop to choke
Sometimes when processing lots of logs I get the following errors. Please
do you have any tips on where/ how i can find out the file / line that is
causing sqoop to choke?
Thank you

2013-03-15 21:01:53,619 INFO org.apache.hadoop.util.NativeCodeLoader
(main): Loaded the native-hadoop library
2013-03-15 21:01:54,301 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl (main): Source name ugi
already exists!
2013-03-15 21:01:54,562 INFO org.apache.hadoop.mapred.MapTask (main): Host
name: domU-12-31-39-02-61-87.compute-1.internal
2013-03-15 21:01:54,618 INFO org.apache.hadoop.util.ProcessTree (main):
setsid exited with exit code 0
2013-03-15 21:01:54,651 INFO org.apache.hadoop.mapred.Task (main):  Using
ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1b7ae22
2013-03-15 21:01:55,123 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader
(main): Loaded native gpl library
2013-03-15 21:01:55,167 WARN com.hadoop.compression.lzo.LzoCodec (main):
Could not find build properties file with revision hash
2013-03-15 21:01:55,167 INFO com.hadoop.compression.lzo.LzoCodec (main):
Successfully loaded & initialized native-lzo library [hadoop-lzo rev
UNKNOWN]
2013-03-15 21:01:55,178 WARN
org.apache.hadoop.io.compress.snappy.LoadSnappy (main): Snappy native
library is available
2013-03-15 21:01:55,178 INFO
org.apache.hadoop.io.compress.snappy.LoadSnappy (main): Snappy native
library loaded
2013-03-15 21:01:59,652 DEBUG
org.apache.sqoop.mapreduce.AsyncSqlOutputFormat (Thread-13): Committing
transaction of 100 statements
2013-03-15 21:01:59,874 DEBUG org.apache.sqoop.mapreduce.AutoProgressMapper
(main): Instructing auto-progress thread to quit.
2013-03-15 21:01:59,875 INFO org.apache.sqoop.mapreduce.AutoProgressMapper
(Thread-14): Auto-progress thread is finished. keepGoing=false
2013-03-15 21:01:59,875 DEBUG org.apache.sqoop.mapreduce.AutoProgressMapper
(main): Waiting for progress thread shutdown...
2013-03-15 21:01:59,875 DEBUG org.apache.sqoop.mapreduce.AutoProgressMapper
(main): Progress thread shutdown detected.
2013-03-15 21:01:59,878 INFO org.apache.hadoop.mapred.TaskLogsTruncater
(main): Initializing logs' truncater with mapRetainSize=-1 and
reduceRetainSize=-1
2013-03-15 21:01:59,958 INFO org.apache.hadoop.io.nativeio.NativeIO (main):
Initialized cache for UID to User mapping with a cache timeout of 14400
seconds.
2013-03-15 21:01:59,958 INFO org.apache.hadoop.io.nativeio.NativeIO (main):
Got UserName hadoop for UID 106 from the native implementation
2013-03-15 21:01:59,961 WARN org.apache.hadoop.mapred.Child (main): Error
running child
java.util.NoSuchElementException
at java.util.AbstractList$Itr.next(AbstractList.java:350)
at
ml_ys_log_gmt_daily_experiment_2.__loadFromFields(ml_ys_log_gmt_daily_experiment_2.java:1290)
at
ml_ys_log_gmt_daily_experiment_2.parse(ml_ys_log_gmt_daily_experiment_2.java:1144)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:77)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:36)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at
org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:182)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:375)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
2013-03-15 21:01:59,976 INFO org.apache.hadoop.mapred.Task (main): Runnning
cleanup for the task
+
Jarek Jarcec Cecho 2013-03-17, 14:37
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB