Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Hive does not package a custom inputformat in the MR job jar when the custom inputformat class is add as aux jar.


+
Mitesh Peshave 2013-07-18, 04:14
+
Andrew Trask 2013-07-18, 04:39
Copy link to this message
-
Re: Hive does not package a custom inputformat in the MR job jar when the custom inputformat class is add as aux jar.
that is what I search for a long time, and no responses. But if you are not
in the cloud (AWS, Azure,...) you can add the jar for your all Datanodes in
$HADOOP_HOME/lib ,  and then restart  the service mapreduce-tasktracker like
this

/etc/init.d/*mapreduce-tasktracker stop

/etc/init.d/*mapreduce-tasktracker start

Hope this help you ;)
2013/7/18 Andrew Trask <[EMAIL PROTECTED]>

> Put them in hive's lib folder?
>
> Sent from my Rotary Phone
>
> On Jul 17, 2013, at 11:14 PM, Mitesh Peshave <[EMAIL PROTECTED]> wrote:
>
> > Hello,
> >
> > I am trying to use a custom inputformat for a hive table.
> >
> > When I add the jar containing the custom inputformat through a client,
> such as the beeline, executing "add jar" command, all seems to work fine.
> In this scenario, hive seems to pass inputformat class to the JT and TTs. I
> believe, it correctly adds the jar to the distributed cache, and the MR
> jobs complete without any errors.
> >
> > But when I add the jar containing the custom input format under hive
> auxlibs diror the hive lib dir, hive does not seem to pass the inputformat
> class to the JT and TTs, causing the MR jobs to fails with
> ClassNotFoundException.
> >
> > The use-case I am looking at here is, multiple users connecting to the
> HiveServer using hive clients and query a table that uses the a custom
> inputformat. I would not want each user to add the jar executing the "add
> jar" command before the users start querying the table.
> >
> > Is there a way to add extra jars to the hive server once and force the
> server to push these jars to JT for every MR jobs it generates?
> >
> > Appreciate,
> > Mitesh
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB