Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - MapReduce - Libjars


Copy link to this message
-
Re: MapReduce - Libjars
Jagat Singh 2012-06-07, 05:38
Hello Karan,

Did you read this article

http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/

You can have external jars in lib directory in your jar while packaging
your jar
or you can use DistributedCache<http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/filecache/DistributedCache.html>
.

The syntax for doing with libjars is something like this

hadoop jar myjar.jar com.driver.ClassName -libjars
/home/thirdparty/grearJar.jar

It would be available at ${mapred.local.dir}/taskTracker/archive/${user.name
}/distcache/

Read that Cloudera article it would give you idea about number of ways of
doing it.

Besides this one previous discussion can also be helpful.
http://search-hadoop.com/m/gKLlP1hMjp21

Regards,

Jagat Singh

On Thu, Jun 7, 2012 at 10:36 AM, <[EMAIL PROTECTED]> wrote:

> Hi,
>
> Within my map reduce programs; I am using an external JAVA library to help
> parse my raw files. When I am submitting my map reduce program, I am
> getting errors as the external class being referenced is not identified.
> Later, I explicitly specified the external jar being referenced with the
> "libjars" option. But I still continue to face problems as below. Any
> inputs to resolve the issue will be helpful.
>
> 12/06/06 13:36:37 ERROR security.UserGroupInformation:
> PriviledgedActionException as:hadoop
> cause:org.apache.hadoop.mapred.InvalidInputException: Input path does not
> exist: hdfs://ldtdsrbd02.etf.barcapetf.com:8020/user/hadoop/-libjars
> Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException:
> Input path does not exist: hdfs://
> ldtdsrbd02.etf.barcapetf.com:8020/user/hadoop/-libjars
>
> Regards,
> Karan
>
>
>
>
> This e-mail and any attachments are confidential and intended
> solely for the addressee and may also be privileged or exempt from
> disclosure under applicable law. If you are not the addressee, or
> have received this e-mail in error, please notify the sender
> immediately, delete it from your system and do not copy, disclose
> or otherwise act upon any part of this e-mail or its attachments.
>
> Internet communications are not guaranteed to be secure or
> virus-free.
> The Barclays Group does not accept responsibility for any loss
> arising from unauthorised access to, or interference with, any
> Internet communications by any third party, or from the
> transmission of any viruses. Replies to this e-mail may be
> monitored by the Barclays Group for operational or business
> reasons.
>
> Any opinion or other information in this e-mail or its attachments
> that does not relate to the business of the Barclays Group is
> personal to the sender and is not given or endorsed by the Barclays
> Group.
>
> Barclays Bank PLC. Registered in England and Wales (registered no.
> 1026167).
> Registered Office: 1 Churchill Place, London, E14 5HP, United
> Kingdom.
>
> Barclays Bank PLC is authorised and regulated by the Financial
> Services Authority.
>