Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> Import data to HDFS using Sqoop2


+
陳彥廷 2013-09-03, 10:03
+
Abraham Elmahrek 2013-09-03, 17:46
+
陳彥廷 2013-09-04, 01:06
+
Abraham Elmahrek 2013-09-04, 01:23
+
Abraham Elmahrek 2013-09-04, 01:24
Copy link to this message
-
Re: Import data to HDFS using Sqoop2
However, with Sqoop 1.4.4, all schema and table names are lower-case, and
it is successful to make import.

As a result, I am not sure if case sensitivity will be a problem in Sqoop 2.

*This is the connection I created:*
1 connection(s) to show:
Connection with id 1 and name ds_msg_log (Created 9/2/13 10:54 AM, Updated
9/3/13 6:35 PM)
Using Connector id 1
  Connection configuration
    JDBC Driver Class: org.postgresql.Driver
    JDBC Connection String: jdbc:postgresql://MY_IP:MY_PORT/invoice
    Username: hadoop
    Password:
    JDBC Connection Properties:
  Security related configuration options
    Max connections: 123

*This is the job I created:*
Job with id 3 and name ds_msg_log (Created 9/2/13 11:37 AM, Updated 9/3/13
5:57 PM)
Using Connection id 1 and Connector id 1
  Database configuration
    Schema name: invoice
    Table name: ds_msg_log
    Table SQL statement:
    Table column names:
    Partition column name:
    Boundary query:
  Output configuration
    Storage type: HDFS
    Output format: TEXT_FILE
    Output directory: /user/root/ds_msg_log_sqoop2
  Throttling resources
    Extractors:
    Loaders:

*The command I use to start this job is:*
submission start --jid 3
On Wed, Sep 4, 2013 at 9:24 AM, Abraham Elmahrek <[EMAIL PROTECTED]> wrote:

> Sorry, I missed that you are using PostGreSQL. I think case sensitivity
> will be a problem with that database.
>
>
> On Tue, Sep 3, 2013 at 6:23 PM, Abraham Elmahrek <[EMAIL PROTECTED]> wrote:
>
>> What database are you using? Are table names or schema names case
>> sensitive in your database? Sqoop quotes all names, so case sensitivity
>> makes a difference here.
>>
>> Also, could you provide the corresponding Sqoop 1.4.4 command that worked
>> for you?
>>
>> -Abe
>>
>>
>> On Tue, Sep 3, 2013 at 6:06 PM, 陳彥廷(Yanting Chen) <[EMAIL PROTECTED]
>> > wrote:
>>
>>> I am pretty sure that the schema "invoice" exists because I successfully
>>> imported data to this schema using Sqoop 1.4.4
>>>
>>> Also, I already remove "*" when creating job.
>>>
>>> The following is the table in invoice schema:
>>>     TABLE DS_MSG_LOG
>>>     (
>>>         MESSAGE_ID VARCHAR2(23) NOT NULL,
>>>         LOGIN_ID VARCHAR2(30),
>>>         ACPT_DTS TIMESTAMP(6),
>>>         DLVR_DTS TIMESTAMP(6),
>>>         SENDER_ID VARCHAR2(30),
>>>         SENDER_VAC_ID VARCHAR2(39),
>>>         RECEIVER_ID VARCHAR2(30),
>>>         RECEIVER_VAC_ID VARCHAR2(39),
>>>         STATUS VARCHAR2(1),
>>>         MESSAGE_TYPE VARCHAR2(8),
>>>         FLOW_TYPE VARCHAR2(5),
>>>         SERVICE_TYPE VARCHAR2(1),
>>>         SOURCE_FILE_NAME VARCHAR2(150),
>>>         ARCHIVE_FILE_NAME VARCHAR2(250),
>>>         ARCHIVE_CHAR_COUNT NUMBER,
>>>         DECRYPT_FILE_NAME VARCHAR2(250),
>>>         DECRYPT_CHAR_COUNT NUMBER,
>>>         RESP_FILE_NAME VARCHAR2(250),
>>>         RESP_CHAR_COUNT NUMBER,
>>>         RESP_FLAG VARCHAR2(1),
>>>         RTG_SEQ VARCHAR2(8),
>>>         RESENT_FLAG VARCHAR2(1) DEFAULT 'N',
>>>         TOTAL_INV_COUNT NUMBER,
>>>         CONSTRAINT PK_DS_MSG_LOG PRIMARY KEY (MESSAGE_ID)
>>>     )
>>>
>>>
>>> On Wed, Sep 4, 2013 at 1:46 AM, Abraham Elmahrek <[EMAIL PROTECTED]>wrote:
>>>
>>>> Hey User,
>>>>
>>>> It looks like the schema "invoice" does not exist in your database.
>>>> Could you please provide your database schema? Also, "*" is unnecessary
>>>> when specifying "Table column names". If you leave it blank it will import
>>>> all columns by default.
>>>>
>>>> -Abe
>>>>
>>>>
>>>> On Tue, Sep 3, 2013 at 3:03 AM, 陳彥廷(Yanting Chen) <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> According to the official guide,
>>>>> http://sqoop.apache.org/docs/1.99.2/Sqoop5MinutesDemo.html , I
>>>>> successfully created a job.
>>>>>
>>>>> However, when I executed the command, submission start --jid 1, I got
>>>>> this error message: "Exception has occurred during processing command
>>>>> Server has returned exception: Exception: java.lang.Throwable Message:
>>>>> GENERIC_JDBC_CONNECTOR_0002:Unable to execute the SQL statement"
+
Abraham Elmahrek 2013-09-04, 22:11
+
Yanting Chen 2013-09-05, 01:37
+
Mengwei Ding 2013-09-05, 02:17
+
Yanting Chen 2013-09-05, 02:23
+
Mengwei Ding 2013-09-05, 02:32
+
Yanting Chen 2013-09-05, 02:35
+
Abraham Elmahrek 2013-09-05, 02:33
+
Yanting Chen 2013-09-05, 02:36
+
Abraham Elmahrek 2013-09-05, 17:55
+
Yanting Chen 2013-09-06, 01:02
+
Yanting Chen 2013-09-09, 07:18
+
Abraham Elmahrek 2013-09-09, 16:22
+
Yanting Chen 2013-09-10, 09:19
+
Abraham Elmahrek 2013-09-13, 18:09
+
Yanting Chen 2013-09-10, 07:32
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB