ZK is showing its runtime JVM classpath (from the JVM that invoked a
ZK client), and not the ZK server's classpath. The path reported is
the below, which is part of /opt/hadoop/lib itself:

/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar

Check for this jar's existence on all nodes and replace them with your
required version perhaps? I'm not sure if it will work but that seems
to be what you wish to try.

I'd instead generally recommend passing the dependencies you need as
part of your job along with it via -libjars and such, and setting the
MR option to have it take precedence over the hadoop-provided jars of
the same kind.

On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <[EMAIL PROTECTED]> wrote:
> Hello,
>
> I am getting the following syslog while running a mapreduce job on my
> cluster:
>
> 2013-03-19 11:00:37,465 INFO
> org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http client,
> config:
> 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
> Initialized cache for UID to User mapping with a cache timeout of 14400
> seconds.
> 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
> UserName hadoop for UID 8004 from the native implementation
> 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error running
> child : java.lang.NoSuchMethodError:
> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
> <init>()V not found
> at
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
> at
> org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
>
>
> I think the issue is that somewhere I have a wrong version of the httpclient
> jar (I need 4.0 or higher but I have a lower version somewhere). I say this
> because the zookeeper classpath in syslog shows a library that I can't find
> it in my cluster anywhere.
>
> Syslog showed the zookeeper classpath as follows:
>
>
> 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.class.path=/opt/hadoop/bin/../conf:
> /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
> /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
> /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
> /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
> /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
> /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
> /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
> /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
> /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
> /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
> /opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
> /opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
> /opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
> /opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
> /opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
> /opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
> /opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
> /opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:

Harsh J

So I tried the replace commons-httpclient.jar with httpclient.jar on all
nodes method and couldn't find a single node that had commons-httpclient
which is why its very confusing that the zookeeper still shows that on the
classpath.. like its cached somewhere? If it is, how do I flush it?

But I like your option of:

I'd instead generally recommend passing the dependencies you need as
part of your job along with it via -libjars and such, and setting the
MR option to have it take precedence over the hadoop-provided jars of
the same kind.

... How do I do this?

Thanks Harsh.
On Tue, Mar 19, 2013 at 12:19 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> ZK is showing its runtime JVM classpath (from the JVM that invoked a
> ZK client), and not the ZK server's classpath. The path reported is
> the below, which is part of /opt/hadoop/lib itself:
>
> /opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
>
> Check for this jar's existence on all nodes and replace them with your
> required version perhaps? I'm not sure if it will work but that seems
> to be what you wish to try.
>
> I'd instead generally recommend passing the dependencies you need as
> part of your job along with it via -libjars and such, and setting the
> MR option to have it take precedence over the hadoop-provided jars of
> the same kind.
>
> On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <[EMAIL PROTECTED]> wrote:
> > Hello,
> >
> > I am getting the following syslog while running a mapreduce job on my
> > cluster:
> >
> > 2013-03-19 11:00:37,465 INFO
> > org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http
> client,
> > config:
> > 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> > Initializing logs' truncater with mapRetainSize=-1 and
> reduceRetainSize=-1
> > 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
> > Initialized cache for UID to User mapping with a cache timeout of 14400
> > seconds.
> > 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
> > UserName hadoop for UID 8004 from the native implementation
> > 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error
> running
> > child : java.lang.NoSuchMethodError:
> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
> > <init>()V not found
> >       at
> >
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
> >       at
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
> >
> >
> > I think the issue is that somewhere I have a wrong version of the
> httpclient
> > jar (I need 4.0 or higher but I have a lower version somewhere). I say
> this
> > because the zookeeper classpath in syslog shows a library that I can't
> find
> > it in my cluster anywhere.
> >
> > Syslog showed the zookeeper classpath as follows:
> >
> >
> > 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
> > environment:java.class.path=/opt/hadoop/bin/../conf:
> >
> /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
> >
> /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
> >
> /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
> >
> /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
> >
> /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
> >
> /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
> >
> /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
> >
> /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
> >
> /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
> >
> /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:

It shouldn't be cached anywhere but try roll-restarting your
tasktrackers after having done the replacement.

On Tue, Mar 19, 2013 at 9:55 PM, Aji Janis <[EMAIL PROTECTED]> wrote:
> So I tried the replace commons-httpclient.jar with httpclient.jar on all
> nodes method and couldn't find a single node that had commons-httpclient
> which is why its very confusing that the zookeeper still shows that on the
> classpath.. like its cached somewhere? If it is, how do I flush it?
>
> But I like your option of:
>
> I'd instead generally recommend passing the dependencies you need as
> part of your job along with it via -libjars and such, and setting the
> MR option to have it take precedence over the hadoop-provided jars of
> the same kind.
>
> ... How do I do this?
>
> Thanks Harsh.
>
>
> On Tue, Mar 19, 2013 at 12:19 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>> ZK is showing its runtime JVM classpath (from the JVM that invoked a
>> ZK client), and not the ZK server's classpath. The path reported is
>> the below, which is part of /opt/hadoop/lib itself:
>>
>> /opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
>>
>> Check for this jar's existence on all nodes and replace them with your
>> required version perhaps? I'm not sure if it will work but that seems
>> to be what you wish to try.
>>
>> I'd instead generally recommend passing the dependencies you need as
>> part of your job along with it via -libjars and such, and setting the
>> MR option to have it take precedence over the hadoop-provided jars of
>> the same kind.
>>
>> On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <[EMAIL PROTECTED]> wrote:
>> > Hello,
>> >
>> > I am getting the following syslog while running a mapreduce job on my
>> > cluster:
>> >
>> > 2013-03-19 11:00:37,465 INFO
>> > org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http
>> > client,
>> > config:
>> > 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> > Initializing logs' truncater with mapRetainSize=-1 and
>> > reduceRetainSize=-1
>> > 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
>> > Initialized cache for UID to User mapping with a cache timeout of 14400
>> > seconds.
>> > 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
>> > UserName hadoop for UID 8004 from the native implementation
>> > 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error
>> > running
>> > child : java.lang.NoSuchMethodError:
>> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
>> > <init>()V not found
>> >       at
>> >
>> > org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
>> >       at
>> >
>> > org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
>> >
>> >
>> > I think the issue is that somewhere I have a wrong version of the
>> > httpclient
>> > jar (I need 4.0 or higher but I have a lower version somewhere). I say
>> > this
>> > because the zookeeper classpath in syslog shows a library that I can't
>> > find
>> > it in my cluster anywhere.
>> >
>> > Syslog showed the zookeeper classpath as follows:
>> >
>> >
>> > 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
>> > environment:java.class.path=/opt/hadoop/bin/../conf:
>> >
>> > /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:

Harsh J

Depending on the version of Hadoop in use, you'd either set
mapreduce.task.classpath.user.precedence,
mapreduce.user.classpath.first or mapreduce.job.user.classpath.first
in your Job's configuration as true before submitting the job.

This feature/parameter is not available in the Apache Hadoop 0.20.2
version, however. It was added in later so you'll have better luck
trying a more recent version.

On Tue, Mar 19, 2013 at 9:55 PM, Aji Janis <[EMAIL PROTECTED]> wrote:
> So I tried the replace commons-httpclient.jar with httpclient.jar on all
> nodes method and couldn't find a single node that had commons-httpclient
> which is why its very confusing that the zookeeper still shows that on the
> classpath.. like its cached somewhere? If it is, how do I flush it?
>
> But I like your option of:
>
> I'd instead generally recommend passing the dependencies you need as
> part of your job along with it via -libjars and such, and setting the
> MR option to have it take precedence over the hadoop-provided jars of
> the same kind.
>
> ... How do I do this?
>
> Thanks Harsh.
>
>
> On Tue, Mar 19, 2013 at 12:19 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>> ZK is showing its runtime JVM classpath (from the JVM that invoked a
>> ZK client), and not the ZK server's classpath. The path reported is
>> the below, which is part of /opt/hadoop/lib itself:
>>
>> /opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
>>
>> Check for this jar's existence on all nodes and replace them with your
>> required version perhaps? I'm not sure if it will work but that seems
>> to be what you wish to try.
>>
>> I'd instead generally recommend passing the dependencies you need as
>> part of your job along with it via -libjars and such, and setting the
>> MR option to have it take precedence over the hadoop-provided jars of
>> the same kind.
>>
>> On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <[EMAIL PROTECTED]> wrote:
>> > Hello,
>> >
>> > I am getting the following syslog while running a mapreduce job on my
>> > cluster:
>> >
>> > 2013-03-19 11:00:37,465 INFO
>> > org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http
>> > client,
>> > config:
>> > 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> > Initializing logs' truncater with mapRetainSize=-1 and
>> > reduceRetainSize=-1
>> > 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
>> > Initialized cache for UID to User mapping with a cache timeout of 14400
>> > seconds.
>> > 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
>> > UserName hadoop for UID 8004 from the native implementation
>> > 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error
>> > running
>> > child : java.lang.NoSuchMethodError:
>> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
>> > <init>()V not found
>> >       at
>> >
>> > org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
>> >       at
>> >
>> > org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
>> >
>> >
>> > I think the issue is that somewhere I have a wrong version of the
>> > httpclient
>> > jar (I need 4.0 or higher but I have a lower version somewhere). I say
>> > this
>> > because the zookeeper classpath in syslog shows a library that I can't
>> > find
>> > it in my cluster anywhere.
>> >
>> > Syslog showed the zookeeper classpath as follows:
>> >
>> >
>> > 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
>> > environment:java.class.path=/opt/hadoop/bin/../conf:
>> >
>> > /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
>> >
>
Harsh J