Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Exec hadoop from Java, reuse JVM (client-side)?


Copy link to this message
-
Exec hadoop from Java, reuse JVM (client-side)?
I know there is a lot of discussion about JVM reuse in Hadoop, but that usually refers to mappers running on the cluste nodesr.  I have a much different question.  I am running a Java program which at one point execs hadoop and that call sometimes fails in the fashion shown below.  Thus, this issue occurs entirely within the client machine (of course, I am currently running in pseudo-distributed mode which convolutes that point somewhat).  In other words, I successfully ran a Java program, but it failed to subsequently run *another* Java program (hadoop).  My interpretation of the hadoop startup scripts (the hadoop command itself for example) is that they run a second JVM in my scenario, and that they fail to allocate enough memory.

Is there any way to run hadoop from within a JVM such that it reuses the local JVM?

EXCEPTION: java.io.IOException: Cannot run program "hadoop": java.io.IOException: error=12, Cannot allocate memory
    java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
    java.lang.Runtime.exec(Runtime.java:593)
    java.lang.Runtime.exec(Runtime.java:466)
    com.util.Shell.run(Shell.java:44)
    com.exe.Foo.bar(Foo.java:107)
    com.exe.Foo.run(Foo.java:205)
    com.exe.Foo.main(Foo.java:227)
Exception in thread "main" java.io.IOException: Cannot run program "hadoop": java.io.IOException: error=12, Cannot allocate memory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at java.lang.Runtime.exec(Runtime.java:593)
at java.lang.Runtime.exec(Runtime.java:466)
at com.util.Shell.run(Shell.java:44)
at com.exe.Foo.bar(Foo.java:107)
at com.exe.Foo.run(Foo.java:205)
at com.exe.Foo.main(Foo.java:227)
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 6 more

________________________________________________________________________________
Keith Wiley     [EMAIL PROTECTED]     keithwiley.com    music.keithwiley.com

"You can scratch an itch, but you can't itch a scratch. Furthermore, an itch can
itch but a scratch can't scratch. Finally, a scratch can itch, but an itch can't
scratch. All together this implies: He scratched the itch from the scratch that
itched but would never itch the scratch from the itch that scratched."
                                           --  Keith Wiley
________________________________________________________________________________
+
Dhruv 2012-08-01, 22:34
+
Keith Wiley 2012-08-01, 23:37
+
Jim Donofrio 2012-08-02, 00:57
+
Keith Wiley 2012-08-02, 05:45
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB