Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> Sqoop import Format Issue


+
Vineet Mishra 2013-01-18, 11:20
+
Vineet Mishra 2013-01-21, 11:20
+
Jarek Jarcec Cecho 2013-01-22, 00:49
+
Vineet Mishra 2013-01-21, 07:31
Copy link to this message
-
Re: Sqoop import Format Issue
Seems like a data type mismatch. E.g. Trying to insert char data in numeric
column!

It could also mean that you are specifying wrong delim.

Can you share 2 sample rows from data on hdfs and table schema?

Thanks,
Abhijeet
On Jan 21, 2013 1:02 PM, "Vineet Mishra" <[EMAIL PROTECTED]> wrote:

> Hallo Everyone,
>
> I am working with the *Sqoop Export* command and while I was trying to
> export the data from HDFS to PostgreSQL Database using the command,
> *
> sudo -u hdfs sqoop export --connect jdbc:postgresql://localhost/mydatabase
> --username vineet -P -m 1 --table testsqoop --export-dir
> /user/hdfs/Sqoop/Sqooptest --input-fields-terminated-by '\0001'*
>
> I was getting following error,
>
> 13/01/21 08:18:23 INFO manager.SqlManager: Using default fetchSize of 1000
> 13/01/21 08:18:23 INFO tool.CodeGenTool: Beginning code generation
> 13/01/21 08:18:23 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM "testsqoop" AS t LIMIT 1
> 13/01/21 08:18:23 INFO orm.CompilationManager: HADOOP_HOME is
> /usr/lib/hadoop/libexec/..
> 13/01/21 08:18:23 INFO orm.CompilationManager: Found hadoop core jar at:
> /usr/lib/hadoop/libexec/../hadoop-core.jar
> Note:
> /tmp/sqoop-hdfs/compile/32830bc954c996cde42e46f6a8599883/sqooptest.java
> uses or overrides a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 13/01/21 08:18:23 ERROR orm.CompilationManager: Could not make directory:
> /home/serendio/.
> 13/01/21 08:18:23 INFO orm.CompilationManager: Writing jar file:
> /tmp/sqoop-hdfs/compile/32830bc954c996cde42e46f6a8599883/sqooptest.jar
> 13/01/21 08:18:23 INFO mapreduce.ExportJobBase: Beginning export of
> testsqoop
> 13/01/21 08:18:25 INFO input.FileInputFormat: Total input paths to process
> : 1
> 13/01/21 08:18:25 INFO input.FileInputFormat: Total input paths to process
> : 1
> 13/01/21 08:18:25 INFO mapred.JobClient: Running job: job_201301150939_0042
> 13/01/21 08:18:26 INFO mapred.JobClient:  map 0% reduce 0%
> 13/01/21 08:18:36 INFO mapred.JobClient: Task Id :
> attempt_201301150939_0042_m_000000_0, Status : FAILED
> java.lang.NumberFormatException
>     at java.math.BigDecimal.<init>(BigDecimal.java:459)
>     at java.math.BigDecimal.<init>(BigDecimal.java:728)
>     at sqooptest.__loadFromFields(sqooptest.java:191)
>     at sqooptest.parse(sqooptest.java:143)
>     at
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:77)
>     at
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:36)
>     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>     at
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:182)
>     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>     at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1122)
>     at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
>
> I even googled but could not find any relevant solution, has anybody
> worked on this issue?
>
> Thanks in advance!
>
>
>
> On Fri, Jan 18, 2013 at 4:50 PM, Vineet Mishra <[EMAIL PROTECTED]
> > wrote:
>
>> Hallo All,
>>
>> I am working with Sqoop to query from the database to for my Map/Reduce
>> job, the issue I am struggling with is
>>
>> 1) Is there a way to directly pass the Sqoop Import Result to the Map for
>> processing rather than storing it to file system and then processing.
>>
>> If above can't be done then let me know whether,
>>
>> 2) I want the format of the file which is coming from the sqoop import
>> command  to be comma(,) separated single line file, but my output is coming
>> as a multi row file.
>>
>>
>> I am querying from the Database fetching a single column of some random
>> values, which is coming in the format,
+
Vineet Mishra 2013-01-21, 08:43
+
abhijeet gaikwad 2013-01-21, 11:53
+
Vineet Mishra 2013-01-21, 12:01
+
Vineet Mishra 2013-01-18, 12:31
+
abhijeet gaikwad 2013-01-18, 12:41
+
abhijeet gaikwad 2013-01-18, 11:50
+
Vineet Mishra 2013-01-18, 12:06
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB