Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> Sqoop exporting error


Copy link to this message
-
Sqoop exporting error
Hi

Sometimes sqoop craps out. I am looking for any tips or help. Thank you
very much !

The following command runs

~/$SQOOP_ROOT/bin/sqoop export --connect jdbc:postgresql://
ec2-XX-XX-XXX-XX.compute-1.amazonaws.com:5662/xxxxxxxxx --username xxxxxxxx
--password xxxxxxxx --table ml_ys_log_gmt --export-dir
$HADOOP_INPUT/$LOGS_HOME/dt=$(date +%Y-%m-%d)
--input-fields-terminated-by='\t' --lines-terminated-by='\n' --verbose
--batch

on the following line

2012-10-27 23:44:25.699 1351381465699
remote-addr=10.119.81.125&user-agent=SAMSUNG-SGH-I637%2FUCIJ2+Profile%2FMIDP-2.1+Configuration%2FCLDC-1.1+Mozilla%2F4.0+%28compatible%3B+MSIE+6.0%3B+Windows+CE%3B+IEMobile+8.12%3B+MSIEMobile+6.0%29
  hxCA    MLNL    1006    10014   1410            kUQN    10014
      click                                           SGH-i637
                   Samsung                 320

I get the error below.

The insert craps out on the second column 'ts' in INSERT INTO ml_ys_log_gmt
(date, ts

The value passed is '[B@6127da' when it should be 1351381465699 (second
field in the tab delimited line above)

Here is the description in the table
ts                     | bigint                   |
        | plain    |
12/10/28 06:57:20 INFO mapred.JobClient: Task Id :
attempt_201210280647_0004_m_000000_0, Status : FAILED
java.io.IOException: java.sql.BatchUpdateException: Batch entry 59 INSERT
INTO ml_ys_log_gmt (date, ts, environment, resource, network, advertiser,
campaign, creative, cost, click, flavor, ui_element_id, ui_element_type,
event, charge_id, charge_type, charge_vertical, charge_payer,
charge_amount, model, imageformatsupport, inputmodesupport, scriptsupport,
vendor, stylesheetsupport, markupsupport, displaywidth, cookiesupport,
displayheight, version, inputdevices, displaycolordepth, device_os,
device_os_version, mobile_browser, mobile_browser_version, is_tablet,
dual_orientation, marketing_name) VALUES ('2012-10-27 23:44:25.699000
+00:00:00', '[B@6127da',
'remote-addr=10.119.81.125&user-agent=SAMSUNG-SGH-I637%2FUCIJ2+Profile%2FMIDP-2.1+Configuration%2FCLDC-1.1+Mozilla%2F4.0+%28compatible%3B+MSIE+6.0%3B+Windows+CE%3B+IEMobile+8.12%3B+MSIEMobile+6.0%29',
'hxCA', 'MLNL', '1006', '10014', '1410', NULL, 'kUQN', '10014', '', '',
'click', '', '', '', '', NULL, 'SGH-i637', '', '', '', 'Samsung', '', '',
'320', '', '240', '', '', '', 'Windows Mobile OS', '5.0', 'Microsoft Mobile
Explorer', '8.12', 'false', 'false', 'BlackJack') was aborted.  Call
getNextException to see the cause.
at
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:220)
at
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:46)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:646)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:78)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:36)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at
org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:182)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:375)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.sql.BatchUpdateException: Batch entry 59 INSERT INTO
ml_ys_log_gmt (date, ts, environment, resource, network, advertiser,
campaign, creative, cost, click, flavor, ui_element_id, ui_element_type,
event, charge_id, charge_type, charge_vertical, charge_payer,
charge_amount, model, imageformatsupport, inputmodesupport, scriptsupport,
vendor, stylesheetsupport, markupsupport, displaywidth, cookiesupport,
displayheight, version, inputdevices, displaycolordepth, device_os,
device_os_version, mobile_browser, mobile_browser_version, is_tablet,
dual_orientation, marketing_name) VALUES ('2012-10-27 23:44:25.699000
+00:00:00', '[B@6127da',
'remote-addr=10.119.81.125&user-agent=SAMSUNG-SGH-I637%2FUCIJ2+Profile%2FMIDP-2.1+Configuration%2FCLDC-1.1+Mozilla%2F4.0+%28compatible%3B+MSIE+6.0%3B+Windows+CE%3B+IEMobile+8.12%3B+MSIEMobile+6.0%29',
'hxCA', 'MLNL', '1006', '10014', '1410', NULL, 'kUQN', '10014', '', '',
'click', '', '', '', '', NULL, 'SGH-i637', '', '', '', 'Samsung', '', '',
'320', '', '240', '', '', '', 'Windows Mobile OS', '5.0', 'Microsoft Mobile
Explorer', '8.12', 'false', 'false', 'BlackJack') was aborted.  Call
getNextException to see the cause.
at
org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2746)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1887)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2893)
at
org.apache.sqoop.mapreduce.AsyncSqlOutputFormat$AsyncSqlExecThread.run(AsyncSqlOutputFormat.java:228)
Matthieu Labour, Engineering | *Action**X* |
584 Broadway, Suite 1002 – NY, NY 10012
415-994-3480 (m)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB