Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> load a serialized object in hadoop


Copy link to this message
-
Re: load a serialized object in hadoop
Just took a look at the bin/hadoop of your particular version
(http://svn.apache.org/viewvc/hadoop/common/tags/release-0.19.2/bin/hadoop?revision=796970&view=markup).
It looks like that HADOOP_CLIENT_OPTS doesn't work with the jar
command, which is fixed in later version.

So try HADOOP_OPTS=-Xmx1000M bin/hadoop ... instead. It would work
because it just translates to the same java command line that worked
for you :)

__Luke

On Wed, Oct 13, 2010 at 4:18 PM, Shi Yu <[EMAIL PROTECTED]> wrote:
> Hi, I tried the following five ways:
>
> Approach 1: in command line
> HADOOP_CLIENT_OPTS=-Xmx4000m bin/hadoop jar WordCount.jar OOloadtest
>
>
> Approach 2: I added the hadoop-site.xml file with the following element.
> Each time I changed, I stop and restart hadoop on all the nodes.
> ...
> <property>
> <name>HADOOP_CLIENT_OPTS</name>
> <value>-Xmx4000m</value>
> </property>
>
> run the command
> $bin/hadoop jar WordCount.jar OOloadtest
>
> Approach 3: I changed like this
> ...
> <property>
> <name>HADOOP_CLIENT_OPTS</name>
> <value>4000m</value>
> </property>
> ....
>
> Then run the command:
> $bin/hadoop jar WordCount.jar OOloadtest
>
> Approach 4: To make sure, I changed the "m" to numbers, that was
> ...
> <property>
> <name>HADOOP_CLIENT_OPTS</name>
> <value>4000000000</value>
> </property>
> ....
>
> Then run the command:
> $bin/hadoop jar WordCount.jar OOloadtest
>
> All these four approaches come to the same "Java heap space" error.
>
> java.lang.OutOfMemoryError: Java heap space
>        at
> java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:45)
>        at java.lang.StringBuilder.<init>(StringBuilder.java:68)
>        at
> java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:2997)
>        at
> java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2818)
>        at java.io.ObjectInputStream.readString(ObjectInputStream.java:1599)
>        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1320)
>        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
>        at java.util.HashMap.readObject(HashMap.java:1028)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
>        at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1846)
>        at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
>        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
>        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
>        at ObjectManager.loadObject(ObjectManager.java:42)
>        at OOloadtest.main(OOloadtest.java:21)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
>        at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>        at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
>
>
> Approach 5:
> In comparison, I called the Java command directly as follows (there is a
> counter showing how much time it costs if the serialized object is
> successfully loaded):
>
> $java -Xms3G -Xmx3G -classpath
> .:WordCount.jar:hadoop-0.19.2-core.jar:lib/log4j-1.2.15.jar OOloadtest
>
> return:
> object loaded, timing (hms): 0 hour(s) 1 minute(s) 12 second(s)