Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Re: Correct way to unzip locally an archive in Yarn


Copy link to this message
-
Re: Correct way to unzip locally an archive in Yarn
I was using 0.23 and was adding files using the -libjars flag (wanted to
upload some jars which were dependencies for my project) but for some
reason I could never find it in the DistributedCache or would always keep
on getting ClassNotFound on the other side. I took the snippet of code
which does that work when you invoke the hadoop jar command and put it in
my class and it all worked fine. I don't know if the problem was with my
code or if it was a bug. Given that -files is so extensively used, I felt
it could be an issue on my side. In the end I started using 1.1 hadoop and
so completely forgot about diving more deeper but I can definitely revive
it and dig in more.

Hopefully you can try using -libjars too and see if that also is facing a
similar issue since they both are command line switches which should have
almost similar behavior.

Thanks,
Viral

On Tue, Feb 19, 2013 at 10:33 AM, Robert Evans <[EMAIL PROTECTED]> wrote:

> Yes if you can trace this down I would be very interested.  We are running
> 0.23.6 without any issues, but that does not mean that there is not some
> bug in the code that is causing this to happen in your situation.
>
> --Bobby
>
> From: Sebastiano Vigna <[EMAIL PROTECTED]>
> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Date: Saturday, February 16, 2013 8:39 AM
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Re: Correct way to unzip locally an archive in Yarn
>
> I will as soon as I can understand what happens on the cluster (no access
> from home). DistributedCache.getLocalCacheFiles() returns in both cases a
> local name for the zip file uploaded with -files, but locally my unzip code
> works, on the cluster it throws a FileNotFoundException.
>
>
> On 16 February 2013 15:22, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>
>> This could be a bug, mind opening a jira? Thanks.
>>
>> On Feb 16, 2013, at 2:34 AM, Sebastiano Vigna wrote:
>>
>> On 15 February 2013 16:57, Robert Evans <[EMAIL PROTECTED]> wrote:
>>
>>> Are you trying to run a Map/Reduce job or are you writing a new YARN
>>> application?  If it is a MR job, then it should work mostly the same as
>>> before (on 1.x). If you are writing a new YARN application then there is
>>> a
>>> separate Map in the ContainerLaunchContext that you need to fill in.
>>
>>
>> It's a MapReduce job (0.23.6). After two days of useless trials, I'm
>> uploading the zip with -files and I wrote a stub to unzip it manually. I
>> was positively unable to get the archive unzipped *to a local directory* in
>> any way.
>>
>> Unfortunately it works in local but not on the cluster. I have still to
>> discover why. :(
>>
>> Ciao,
>>
>>
>>
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB