Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Hive does not package a custom inputformat in the MR job jar when the custom inputformat class is add as aux jar.


+
Mitesh Peshave 2013-07-18, 04:14
+
Andrew Trask 2013-07-18, 04:39
Copy link to this message
-
Re: Hive does not package a custom inputformat in the MR job jar when the custom inputformat class is add as aux jar.
that is what I search for a long time, and no responses. But if you are not
in the cloud (AWS, Azure,...) you can add the jar for your all Datanodes in
$HADOOP_HOME/lib ,  and then restart  the service mapreduce-tasktracker like
this

/etc/init.d/*mapreduce-tasktracker stop

/etc/init.d/*mapreduce-tasktracker start

Hope this help you ;)
2013/7/18 Andrew Trask <[EMAIL PROTECTED]>

> Put them in hive's lib folder?
>
> Sent from my Rotary Phone
>
> On Jul 17, 2013, at 11:14 PM, Mitesh Peshave <[EMAIL PROTECTED]> wrote:
>
> > Hello,
> >
> > I am trying to use a custom inputformat for a hive table.
> >
> > When I add the jar containing the custom inputformat through a client,
> such as the beeline, executing "add jar" command, all seems to work fine.
> In this scenario, hive seems to pass inputformat class to the JT and TTs. I
> believe, it correctly adds the jar to the distributed cache, and the MR
> jobs complete without any errors.
> >
> > But when I add the jar containing the custom input format under hive
> auxlibs diror the hive lib dir, hive does not seem to pass the inputformat
> class to the JT and TTs, causing the MR jobs to fails with
> ClassNotFoundException.
> >
> > The use-case I am looking at here is, multiple users connecting to the
> HiveServer using hive clients and query a table that uses the a custom
> inputformat. I would not want each user to add the jar executing the "add
> jar" command before the users start querying the table.
> >
> > Is there a way to add extra jars to the hive server once and force the
> server to push these jars to JT for every MR jobs it generates?
> >
> > Appreciate,
> > Mitesh
>