Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> Import data to HDFS using Sqoop2


Copy link to this message
-
Re: Import data to HDFS using Sqoop2
TABLE DS_MSG_LOG
    (
        MESSAGE_ID VARCHAR2(23) NOT NULL,
        LOGIN_ID VARCHAR2(30),
        ACPT_DTS TIMESTAMP(6),
        DLVR_DTS TIMESTAMP(6),
        SENDER_ID VARCHAR2(30),
        SENDER_VAC_ID VARCHAR2(39),
        RECEIVER_ID VARCHAR2(30),
        RECEIVER_VAC_ID VARCHAR2(39),
        STATUS VARCHAR2(1),
        MESSAGE_TYPE VARCHAR2(8),
        FLOW_TYPE VARCHAR2(5),
        SERVICE_TYPE VARCHAR2(1),
        SOURCE_FILE_NAME VARCHAR2(150),
        ARCHIVE_FILE_NAME VARCHAR2(250),
        ARCHIVE_CHAR_COUNT NUMBER,
        DECRYPT_FILE_NAME VARCHAR2(250),
        DECRYPT_CHAR_COUNT NUMBER,
        RESP_FILE_NAME VARCHAR2(250),
        RESP_CHAR_COUNT NUMBER,
        RESP_FLAG VARCHAR2(1),
        RTG_SEQ VARCHAR2(8),
        RESENT_FLAG VARCHAR2(1) DEFAULT 'N',
        TOTAL_INV_COUNT NUMBER,
        CONSTRAINT PK_DS_MSG_LOG PRIMARY KEY (MESSAGE_ID)
    )
On Fri, Sep 6, 2013 at 1:55 AM, Abraham Elmahrek <[EMAIL PROTECTED]> wrote:

> Could you provide your schema from PostGreSQL? Mengwei is likely right.
>
>
> On Wed, Sep 4, 2013 at 7:36 PM, Yanting Chen <[EMAIL PROTECTED]>wrote:
>
>> Actually the schema comes from Oracle. However, I try to modify it to let
>> it fit into PostgreSQL database. So, now the database I use is PostgreSQL.
>>
>>
>> On Thu, Sep 5, 2013 at 10:33 AM, Abraham Elmahrek <[EMAIL PROTECTED]>wrote:
>>
>>> Yanting,
>>>
>>> Also, it seems like the schema you've provided is for an Oracle
>>> database. i.e. VARCHAR2 and NUMBER are datatypes specific to Oracle. Could
>>> you please use an oracle connection string and driver? i.e. oracle.jdbc.driver.OracleDriver
>>> and jdbc:oracle:thin:@host:port:SID.
>>>
>>> -abe
>>>
>>>
>>> On Wed, Sep 4, 2013 at 7:32 PM, Mengwei Ding <[EMAIL PROTECTED]>wrote:
>>>
>>>> Hmm... would you mind showing us your most updated job configuration by
>>>> typing "show job --jid 3"? I just want to make sure that you provide the
>>>> partition column correctly.
>>>>
>>>> Also, I notice that the primary key for this table is "VARCHAR(23)"
>>>> type, this might be the problem.
>>>>
>>>> Mengwei
>>>>
>>>>
>>>> On Wed, Sep 4, 2013 at 10:23 PM, Yanting Chen <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> Hi Mengwei
>>>>>
>>>>> I try to set primary key into partition column. But still the same
>>>>> error!
>>>>>
>>>>>
>>>>> On Thu, Sep 5, 2013 at 10:17 AM, Mengwei Ding <[EMAIL PROTECTED]>wrote:
>>>>>
>>>>>> Hi Yanting,
>>>>>>
>>>>>> It seems like you did not specify the 'partition column' for the job.
>>>>>> Generally, the primary key of  the table is a good choice for 'parition
>>>>>> column'.
>>>>>>
>>>>>> You could use 'update job -jid 3' to update the job configuration.
>>>>>>
>>>>>> Mengwei
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 4, 2013 at 9:37 PM, Yanting Chen <[EMAIL PROTECTED]>wrote:
>>>>>>
>>>>>>> Abraham,
>>>>>>>
>>>>>>> Thanks for you answer. I review my database.
>>>>>>> I think the database name is invoice and the schema name is public,
>>>>>>> just like the picture below.
>>>>>>> http://imgur.com/ns0iNLi
>>>>>>>
>>>>>>> So, I change the schema name with a new value, "public"
>>>>>>> Then, I run this job and get a different error.
>>>>>>>
>>>>>>> Status: FAILURE_ON_SUBMIT
>>>>>>> Creation date: 2013-09-05 09:30:44 CST
>>>>>>> Last update date: 2013-09-05 09:30:44 CST
>>>>>>> Exception: org.apache.sqoop.common.SqoopException:
>>>>>>> GENERIC_JDBC_CONNECTOR_0011:The type is not supported - 12
>>>>>>> Stack trace: org.apache.sqoop.common.SqoopException:
>>>>>>> GENERIC_JDBC_CONNECTOR_0011:The type is not supported - 12
>>>>>>> at
>>>>>>> org.apache.sqoop.connector.jdbc.GenericJdbcImportPartitioner.getPartitions(GenericJdbcImportPartitioner.java:87)
>>>>>>>  at
>>>>>>> org.apache.sqoop.connector.jdbc.GenericJdbcImportPartitioner.getPartitions(GenericJdbcImportPartitioner.java:32)
>>>>>>> at
>>>>>>> org.apache.sqoop.job.mr.SqoopInputFormat.getSplits(SqoopInputFormat.java:71)
>>>>>>>  at
>>>>>>> org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:452)