Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - setting hadoop for pseudo distributed mode.


+
jamal sasha 2012-12-27, 20:03
+
Mohammad Tariq 2012-12-27, 20:49
Copy link to this message
-
Re: setting hadoop for pseudo distributed mode.
jamal sasha 2012-12-27, 21:08
Hi,
  Thanks for throwing insight.

So the code snippet looks like this

String interout = final_output + "/intermediate";

    try {
      new CreateInterOutput().main(new String[] { input, interout });
    } catch (Exception e) {
      e.printStackTrace();
      return;
    }

    try {
    new CreateFinalOutput().main(new String[] { interout, final_output });

    } catch (Exception e) {
      e.printStackTrace();
      return;
    }

Any suggestions where it might be faltering?
On Thu, Dec 27, 2012 at 12:49 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:

> Hello Jamal,
>
>    Please find my commands embedded below :
>
> Q1) How did putting those two lines solved the issue??
> >> By adding those two resources you make sure that your code looks for
> the input path inside HDFS, which would otherwise look for it in the local
> FS by default. The files core-site.xml and hdfs-site.xml tell your code
> where to go for NN and DN.
>
> Q2)
>   I am now using third party libraries which are taking input from hdfs
> and writing output to hdfs...
> But in an intermediatory step, it creates a raw output.. I am again
> getting the error:
>  ERROR security.UserGroupInformation: PriviledgedActionException
> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
> How do I resolve this?
> >>If you are able to compile your code properly then there is no problem
> with the third party libraries which you are using. It looks like to me
> that your code doesn't have the proper info about the intermediate path.
> Please make sure you have told your code the exact location of intermediate
> output.
>
>
> Best Regards,
> Tariq
> +91-9741563634
> https://mtariq.jux.com/
>
>
> On Fri, Dec 28, 2012 at 1:33 AM, jamal sasha <[EMAIL PROTECTED]>wrote:
>
>> Hi,
>>   So I am still in process of learning hadoop.
>> I tried to run wordcount.java (by writing my own mapper reducer..
>> creating jar and then running it in a pseudo distributed mode).
>>
>> At that time I got an error, something like
>>  ERROR security.UserGroupInformation: PriviledgedActionException
>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>> does not exist: hdfs://localhost:54310/user/hduser/wiki
>> So I googled around.. and found that I should put the following two lines
>> on my driver code:
>>
>>     conf.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
>>     conf.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml
>> file"));
>>
>> (path of where my core-site and hdfs-site are) and after that it ran just
>> fine.
>>
>> Q1) How did putting those two lines solved the issue??
>> Q2)
>>   I am now using third party libraries which are taking input from hdfs
>> and writing output to hdfs...
>> But in an intermediatory step, it creates a raw output.. I am again
>> getting the error:
>>  ERROR security.UserGroupInformation: PriviledgedActionException
>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
>> How do I resolve this?
>>
>> Any suggestions.
>> THanks
>> Jamal.
>>
>
>
+
Mohammad Tariq 2012-12-27, 23:10