Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> Sqoop 1.4.2 checkout from trunk (installation problem) -sqoop 1.4.1 incompatible with MSSQL Server Connector


+
Victor Sanchez 2012-06-27, 15:32
+
Cheolsoo Park 2012-06-27, 16:01
+
Cheolsoo Park 2012-06-27, 16:38
+
Victor Sanchez 2012-06-28, 09:49
+
Cheolsoo Park 2012-06-28, 21:19
+
Victor Sanchez 2012-06-29, 08:02
+
Cheolsoo Park 2012-06-30, 02:03
+
Victor Sanchez 2012-07-02, 14:21
Copy link to this message
-
Re: Sqoop 1.4.2 checkout from trunk (installation problem) -sqoop 1.4.1 incompatible with MSSQL Server Connector
Hi Victor,

Thank you for providing all the information. I think that I know what's the
problem.

HADOOP_MAPRED_HOME is set to "hadoop-mapreduce" (MR2) instead of
"hadoop-0.20-mapreduce" (MR1).

HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce
This results in that MR2 jars are added to Sqoop's classpath instead of MR1
jars, and Sqoop ends up calling MR2 Hadoop at runtime. Even if you don't
add yarn to your cluster in CM, both MR1 and MR2 are always installed, and
HADOOP_MAPRED_HOME must point to the one that you're using. So in your
case, it must be set to MR1:

HADOOP_MAPRED_HOME=/usr/lib/hadoop-0.20-mapreduce
Now I am wondering if you have HBase installed on the machine where you're
running Sqoop. If so, that should explain why HADOOP_MAPRED_HOME is set to
MR2. If you look at "/usr/lib/sqoop/bin/configure-sqoop", it invokes "hbase
classpath" if hBase is installed:

if [ -e "$HBASE_HOME/bin/hbase" ]; then
>   TMP_SQOOP_CLASSPATH=${SQOOP_CLASSPATH}:`$HBASE_HOME/bin/hbase classpath`
>   SQOOP_CLASSPATH=${TMP_SQOOP_CLASSPATH}
> fi
 In turn, "/usr/bin/hbase" executes the following line:

. /etc/default/hadoop
And "/etc/default/hadoop" always sets HADOOP_MAPRED_HOME to MR2:

export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce
In fact, this is a packaging bug in CDH4 as far as I am concerned because
regardless whether the user sets HADOOP_MAPRED_HOME to MR1 or MR2, it will
be reset to MR2 by "/usr/bin/hbase". I am going to open an internal bug
report for this.

As a workaround, you could edit the following line in "/etc/default/hadoop"

from

> export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce

to

> export HADOOP_MAPRED_HOME=/usr/lib/hadoop-0.20-mapreduce
If you're going to use MR1 only, this shouldn't cause any problem. Please
let me know if this solves your problem.

Thanks!
Cheolsoo

On Mon, Jul 2, 2012 at 7:21 AM, Victor Sanchez <[EMAIL PROTECTED]>wrote:

>  Hello Cheolsoo,****
>
> ** **
>
> I was trying to dig more into the problem, so far I just noticed that as
> far as I’m using Hadoop version 2.0 there is some changes at least in the
> way you could make calls to the HDFS. Plus there are some extra
> directiories not present in previous versions. So I was wondering if that
> could be the problem.****
>
> ** **
>
> *$ hadoop dfs -ls /*
>
> *DEPRECATED: Use of this script to execute hdfs command is deprecated.*
>
> *Instead use the hdfs command for it.*
>
> ** **
>
> Found 5 items****
>
> drwxrwxrwx   - hue     cloudAD          0 2012-06-28 12:41 /SQL_SQOOP****
>
> …****
>
> ** **
>
> *$ hdfs dfs -ls /*
>
> Found 5 items****
>
> drwxrwxrwx   - hue     cloudAD          0 2012-06-28 12:41 /SQL_SQOOP****
>
> …****
>
> ** **
>
> *$ ls -all /usr/lib/ *| grep hadoop****
>
> drwxr-xr-x  10 root root 4.0K Jun 15 18:42 hadoop/****
>
> *-rw-r--r--   1 root root 1.7M Jun 21 18:11 hadoop-0.20-*
>
> drwxr-xr-x  12 root root 4.0K Jun 15 18:42 hadoop-0.20-mapreduce/****
>
> drwxr-xr-x   7 root root 4.0K Jun 15 18:35 hadoop-hdfs/****
>
> drwxr-xr-x   6 root root 4.0K Jun 15 18:37 hadoop-httpfs/****
>
> drwxr-xr-x   6 root root 4.0K Jun 15 18:39 hadoop-mapreduce/****
>
> drwxr-xr-x   7 root root 4.0K Jun 15 18:39 hadoop-yarn/****
>
> ** **
>
> *$ ls -all /usr/lib/hadoop*
>
> total 3.3M****
>
> drwxr-xr-x  10 root root 4.0K Jun 15 18:42 ./****
>
> dr-xr-xr-x. 40 root root 4.0K Jun 26 14:37 ../****
>
> drwxr-xr-x   2 root root 4.0K Jun 15 18:34 bin/****
>
> drwxr-xr-x   2 root root 4.0K Jun 15 18:42 client/****
>
> drwxr-xr-x   2 root root 4.0K Jun 15 18:42 client-0.20/****
>
> drwxr-xr-x   2 root root 4.0K Jun 15 19:07 cloudera/****
>
> drwxr-xr-x   2 root root 4.0K Jun 15 18:34 etc/****
>
> -rw-r--r--   1 root root  17K Jun  5 02:09
> hadoop-annotations-2.0.0-cdh4.0.0.jar****
>
> lrwxrwxrwx   1 root root   37 Jun 15 18:34 hadoop-annotations.jar ->
> hadoop-annotations-2.0.0-cdh4.0.0.jar****
>
> -rw-r--r--   1 root root  43K Jun  5 02:09 hadoop-auth-2.0.0-cdh4.0.0.jar*
> ***
>
> lrwxrwxrwx   1 root root   30 Jun 15 18:34 hadoop-auth.jar ->
+
Victor Sanchez 2012-07-03, 07:07
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB