Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop, mail # user - Import data to HDFS using Sqoop2


+
陳彥廷 2013-09-03, 10:03
+
Abraham Elmahrek 2013-09-03, 17:46
+
陳彥廷 2013-09-04, 01:06
+
Abraham Elmahrek 2013-09-04, 01:23
+
Abraham Elmahrek 2013-09-04, 01:24
Copy link to this message
-
Re: Import data to HDFS using Sqoop2
Yanting Chen 2013-09-04, 01:34
However, with Sqoop 1.4.4, all schema and table names are lower-case, and
it is successful to make import.

As a result, I am not sure if case sensitivity will be a problem in Sqoop 2.

*This is the connection I created:*
1 connection(s) to show:
Connection with id 1 and name ds_msg_log (Created 9/2/13 10:54 AM, Updated
9/3/13 6:35 PM)
Using Connector id 1
  Connection configuration
    JDBC Driver Class: org.postgresql.Driver
    JDBC Connection String: jdbc:postgresql://MY_IP:MY_PORT/invoice
    Username: hadoop
    Password:
    JDBC Connection Properties:
  Security related configuration options
    Max connections: 123

*This is the job I created:*
Job with id 3 and name ds_msg_log (Created 9/2/13 11:37 AM, Updated 9/3/13
5:57 PM)
Using Connection id 1 and Connector id 1
  Database configuration
    Schema name: invoice
    Table name: ds_msg_log
    Table SQL statement:
    Table column names:
    Partition column name:
    Boundary query:
  Output configuration
    Storage type: HDFS
    Output format: TEXT_FILE
    Output directory: /user/root/ds_msg_log_sqoop2
  Throttling resources
    Extractors:
    Loaders:

*The command I use to start this job is:*
submission start --jid 3
On Wed, Sep 4, 2013 at 9:24 AM, Abraham Elmahrek <[EMAIL PROTECTED]> wrote:

> Sorry, I missed that you are using PostGreSQL. I think case sensitivity
> will be a problem with that database.
>
>
> On Tue, Sep 3, 2013 at 6:23 PM, Abraham Elmahrek <[EMAIL PROTECTED]> wrote:
>
>> What database are you using? Are table names or schema names case
>> sensitive in your database? Sqoop quotes all names, so case sensitivity
>> makes a difference here.
>>
>> Also, could you provide the corresponding Sqoop 1.4.4 command that worked
>> for you?
>>
>> -Abe
>>
>>
>> On Tue, Sep 3, 2013 at 6:06 PM, 陳彥廷(Yanting Chen) <[EMAIL PROTECTED]
>> > wrote:
>>
>>> I am pretty sure that the schema "invoice" exists because I successfully
>>> imported data to this schema using Sqoop 1.4.4
>>>
>>> Also, I already remove "*" when creating job.
>>>
>>> The following is the table in invoice schema:
>>>     TABLE DS_MSG_LOG
>>>     (
>>>         MESSAGE_ID VARCHAR2(23) NOT NULL,
>>>         LOGIN_ID VARCHAR2(30),
>>>         ACPT_DTS TIMESTAMP(6),
>>>         DLVR_DTS TIMESTAMP(6),
>>>         SENDER_ID VARCHAR2(30),
>>>         SENDER_VAC_ID VARCHAR2(39),
>>>         RECEIVER_ID VARCHAR2(30),
>>>         RECEIVER_VAC_ID VARCHAR2(39),
>>>         STATUS VARCHAR2(1),
>>>         MESSAGE_TYPE VARCHAR2(8),
>>>         FLOW_TYPE VARCHAR2(5),
>>>         SERVICE_TYPE VARCHAR2(1),
>>>         SOURCE_FILE_NAME VARCHAR2(150),
>>>         ARCHIVE_FILE_NAME VARCHAR2(250),
>>>         ARCHIVE_CHAR_COUNT NUMBER,
>>>         DECRYPT_FILE_NAME VARCHAR2(250),
>>>         DECRYPT_CHAR_COUNT NUMBER,
>>>         RESP_FILE_NAME VARCHAR2(250),
>>>         RESP_CHAR_COUNT NUMBER,
>>>         RESP_FLAG VARCHAR2(1),
>>>         RTG_SEQ VARCHAR2(8),
>>>         RESENT_FLAG VARCHAR2(1) DEFAULT 'N',
>>>         TOTAL_INV_COUNT NUMBER,
>>>         CONSTRAINT PK_DS_MSG_LOG PRIMARY KEY (MESSAGE_ID)
>>>     )
>>>
>>>
>>> On Wed, Sep 4, 2013 at 1:46 AM, Abraham Elmahrek <[EMAIL PROTECTED]>wrote:
>>>
>>>> Hey User,
>>>>
>>>> It looks like the schema "invoice" does not exist in your database.
>>>> Could you please provide your database schema? Also, "*" is unnecessary
>>>> when specifying "Table column names". If you leave it blank it will import
>>>> all columns by default.
>>>>
>>>> -Abe
>>>>
>>>>
>>>> On Tue, Sep 3, 2013 at 3:03 AM, 陳彥廷(Yanting Chen) <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> According to the official guide,
>>>>> http://sqoop.apache.org/docs/1.99.2/Sqoop5MinutesDemo.html , I
>>>>> successfully created a job.
>>>>>
>>>>> However, when I executed the command, submission start --jid 1, I got
>>>>> this error message: "Exception has occurred during processing command
>>>>> Server has returned exception: Exception: java.lang.Throwable Message:
>>>>> GENERIC_JDBC_CONNECTOR_0002:Unable to execute the SQL statement"
+
Abraham Elmahrek 2013-09-04, 22:11
+
Yanting Chen 2013-09-05, 01:37
+
Mengwei Ding 2013-09-05, 02:17
+
Yanting Chen 2013-09-05, 02:23
+
Mengwei Ding 2013-09-05, 02:32
+
Yanting Chen 2013-09-05, 02:35
+
Abraham Elmahrek 2013-09-05, 02:33
+
Yanting Chen 2013-09-05, 02:36
+
Abraham Elmahrek 2013-09-05, 17:55
+
Yanting Chen 2013-09-06, 01:02
+
Yanting Chen 2013-09-09, 07:18
+
Abraham Elmahrek 2013-09-09, 16:22
+
Yanting Chen 2013-09-10, 09:19
+
Abraham Elmahrek 2013-09-13, 18:09
+
Yanting Chen 2013-09-10, 07:32