Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> Import data to HDFS using Sqoop2


+
陳彥廷 2013-09-03, 10:03
+
Abraham Elmahrek 2013-09-03, 17:46
+
陳彥廷 2013-09-04, 01:06
+
Abraham Elmahrek 2013-09-04, 01:23
+
Abraham Elmahrek 2013-09-04, 01:24
+
Yanting Chen 2013-09-04, 01:34
+
Abraham Elmahrek 2013-09-04, 22:11
+
Yanting Chen 2013-09-05, 01:37
+
Mengwei Ding 2013-09-05, 02:17
+
Yanting Chen 2013-09-05, 02:23
+
Mengwei Ding 2013-09-05, 02:32
+
Yanting Chen 2013-09-05, 02:35
+
Abraham Elmahrek 2013-09-05, 02:33
+
Yanting Chen 2013-09-05, 02:36
+
Abraham Elmahrek 2013-09-05, 17:55
Copy link to this message
-
Re: Import data to HDFS using Sqoop2
TABLE DS_MSG_LOG
    (
        MESSAGE_ID VARCHAR2(23) NOT NULL,
        LOGIN_ID VARCHAR2(30),
        ACPT_DTS TIMESTAMP(6),
        DLVR_DTS TIMESTAMP(6),
        SENDER_ID VARCHAR2(30),
        SENDER_VAC_ID VARCHAR2(39),
        RECEIVER_ID VARCHAR2(30),
        RECEIVER_VAC_ID VARCHAR2(39),
        STATUS VARCHAR2(1),
        MESSAGE_TYPE VARCHAR2(8),
        FLOW_TYPE VARCHAR2(5),
        SERVICE_TYPE VARCHAR2(1),
        SOURCE_FILE_NAME VARCHAR2(150),
        ARCHIVE_FILE_NAME VARCHAR2(250),
        ARCHIVE_CHAR_COUNT NUMBER,
        DECRYPT_FILE_NAME VARCHAR2(250),
        DECRYPT_CHAR_COUNT NUMBER,
        RESP_FILE_NAME VARCHAR2(250),
        RESP_CHAR_COUNT NUMBER,
        RESP_FLAG VARCHAR2(1),
        RTG_SEQ VARCHAR2(8),
        RESENT_FLAG VARCHAR2(1) DEFAULT 'N',
        TOTAL_INV_COUNT NUMBER,
        CONSTRAINT PK_DS_MSG_LOG PRIMARY KEY (MESSAGE_ID)
    )
On Fri, Sep 6, 2013 at 1:55 AM, Abraham Elmahrek <[EMAIL PROTECTED]> wrote:

> Could you provide your schema from PostGreSQL? Mengwei is likely right.
>
>
> On Wed, Sep 4, 2013 at 7:36 PM, Yanting Chen <[EMAIL PROTECTED]>wrote:
>
>> Actually the schema comes from Oracle. However, I try to modify it to let
>> it fit into PostgreSQL database. So, now the database I use is PostgreSQL.
>>
>>
>> On Thu, Sep 5, 2013 at 10:33 AM, Abraham Elmahrek <[EMAIL PROTECTED]>wrote:
>>
>>> Yanting,
>>>
>>> Also, it seems like the schema you've provided is for an Oracle
>>> database. i.e. VARCHAR2 and NUMBER are datatypes specific to Oracle. Could
>>> you please use an oracle connection string and driver? i.e. oracle.jdbc.driver.OracleDriver
>>> and jdbc:oracle:thin:@host:port:SID.
>>>
>>> -abe
>>>
>>>
>>> On Wed, Sep 4, 2013 at 7:32 PM, Mengwei Ding <[EMAIL PROTECTED]>wrote:
>>>
>>>> Hmm... would you mind showing us your most updated job configuration by
>>>> typing "show job --jid 3"? I just want to make sure that you provide the
>>>> partition column correctly.
>>>>
>>>> Also, I notice that the primary key for this table is "VARCHAR(23)"
>>>> type, this might be the problem.
>>>>
>>>> Mengwei
>>>>
>>>>
>>>> On Wed, Sep 4, 2013 at 10:23 PM, Yanting Chen <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> Hi Mengwei
>>>>>
>>>>> I try to set primary key into partition column. But still the same
>>>>> error!
>>>>>
>>>>>
>>>>> On Thu, Sep 5, 2013 at 10:17 AM, Mengwei Ding <[EMAIL PROTECTED]>wrote:
>>>>>
>>>>>> Hi Yanting,
>>>>>>
>>>>>> It seems like you did not specify the 'partition column' for the job.
>>>>>> Generally, the primary key of  the table is a good choice for 'parition
>>>>>> column'.
>>>>>>
>>>>>> You could use 'update job -jid 3' to update the job configuration.
>>>>>>
>>>>>> Mengwei
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 4, 2013 at 9:37 PM, Yanting Chen <[EMAIL PROTECTED]>wrote:
>>>>>>
>>>>>>> Abraham,
>>>>>>>
>>>>>>> Thanks for you answer. I review my database.
>>>>>>> I think the database name is invoice and the schema name is public,
>>>>>>> just like the picture below.
>>>>>>> http://imgur.com/ns0iNLi
>>>>>>>
>>>>>>> So, I change the schema name with a new value, "public"
>>>>>>> Then, I run this job and get a different error.
>>>>>>>
>>>>>>> Status: FAILURE_ON_SUBMIT
>>>>>>> Creation date: 2013-09-05 09:30:44 CST
>>>>>>> Last update date: 2013-09-05 09:30:44 CST
>>>>>>> Exception: org.apache.sqoop.common.SqoopException:
>>>>>>> GENERIC_JDBC_CONNECTOR_0011:The type is not supported - 12
>>>>>>> Stack trace: org.apache.sqoop.common.SqoopException:
>>>>>>> GENERIC_JDBC_CONNECTOR_0011:The type is not supported - 12
>>>>>>> at
>>>>>>> org.apache.sqoop.connector.jdbc.GenericJdbcImportPartitioner.getPartitions(GenericJdbcImportPartitioner.java:87)
>>>>>>>  at
>>>>>>> org.apache.sqoop.connector.jdbc.GenericJdbcImportPartitioner.getPartitions(GenericJdbcImportPartitioner.java:32)
>>>>>>> at
>>>>>>> org.apache.sqoop.job.mr.SqoopInputFormat.getSplits(SqoopInputFormat.java:71)
>>>>>>>  at
>>>>>>> org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:452)
+
Yanting Chen 2013-09-09, 07:18
+
Abraham Elmahrek 2013-09-09, 16:22
+
Yanting Chen 2013-09-10, 09:19
+
Abraham Elmahrek 2013-09-13, 18:09
+
Yanting Chen 2013-09-10, 07:32
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB