Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Externally submitted MapReduce Job Fails at Startup Help Please...


+
Jamal x 2011-10-28, 17:17
Copy link to this message
-
Re: Externally submitted MapReduce Job Fails at Startup Help Please...
Hi,

I always find that using the -libjars command line option is the
easiest way to push jars to the cluster.

Also, you may want to checkout Apache Sqoop:
http://incubator.apache.org/projects/sqoop.html

Brock

On Fri, Oct 28, 2011 at 12:17 PM, Jamal x <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I wrote a small test program to perform a simple database extraction of
> information from a simple table on a remote cluster.  However, it fails to
> execute successfully when I run from eclipse it with the following
> exception:
>
> 12:36:08,993  WARN main mapred.JobClient:659 - Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 12:36:09,567  WARN main mapred.JobClient:776 - No job jar file set.  User
> classes may not be found. See JobConf(Class) or JobConf#setJar(String).
> java.lang.RuntimeException: Error in configuring object
>    at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>    at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>    at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>
>    at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:575)
>    at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
>
>    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
>    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at javax.security.auth.Subject.doAs(Subject.java:396)
>    at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>
>    at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.reflect.InvocationTargetException
>    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
>    at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
>    at java.lang.reflect.Method.invoke(Method.java:597)
>    at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>    ... 11 more
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> com.mysql.jdbc.Driver
>    at
> org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:271)
>
>    ... 16 more
> Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
>    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>    at java.lang.Class.forName0(Native Method)
>    at java.lang.Class.forName(Class.java:169)
>    at
> org.apache.hadoop.mapred.lib.db.DBConfiguration.getConnection(DBConfiguration.java:123)
>
>    at
> org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:266)
>
>    ... 16 more
>
>
> I do have the mysql-connector jar under the $HADOOP_HOME/lib folder on all
> servers in the cluster, and even tried using the
> DistributedCache.addArchiveToClassPath method, with no success.  Can someone
> please help me figure out what is going on here?
>
> Here is my simple main which performs the remote submission of the job:
> public int run(String[] arg0) throws Exception {
>
>        System.out.println("Setting up job configuration....");
>        Configuration conf = new Configuration();
>        conf.set("mapred.job.tracker", "jobtracker.hostname:8021");
>        conf.set("fs.default.name", "hdfs://namenode.hostname:9000");
>        conf.set("keep.failed.task.files", "true");
>        conf.set("mapred.child.java.opts", "-Xmx1024m");
>
>        FileSystem fs = FileSystem.get(conf);
+
Jamal x 2011-10-28, 19:56
+
Steve Lewis 2011-10-29, 20:00
+
Jamal B 2011-11-02, 04:51
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB